BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true`

### Bug Report

# BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true`

**Reporter:** GitHub Copilot
**Date:** 2026-04-27
**Severity:** Medium
**Reproducibility:** Always

---

## Summary

When `model_test` is called with `draft_mode='true'`, any scenario containing a `when t > X`
conditional assessment fails with a simulation setup error, regardless of time format (`1s`,
`0.5s`, `500ms`). The same scenarios pass without error when `draft_mode='false'`.

## Expected Behavior

`draft_mode=true` should produce the same assessment results as `draft_mode=false` for `when t > X`
conditional clauses. All three time formats (`1s`, `0.5s`, `500ms`) should be accepted and evaluated
correctly in both modes.

## Actual Behavior

In `draft_mode=true`, every scenario containing a `when t > X` assessment fails with:

> Unable to evaluate assessments because of errors during test case setup or model simulation

No time format works — integer seconds (`1s`), decimal seconds (`0.5s`), and milliseconds (`500ms`)
all produce the same generic setup/simulation error. In `draft_mode=false`, all three formats succeed.

## Reproduction Steps

**Prerequisites:**
- SATK installed and on the MATLAB path
- MATLAB R2024b

1. In a working directory, run the following MATLAB snippet to build `WindSourceSelector.slx` — a
   Stateflow source-selector chart with mixed port types (`boolean`, `single`, `uint16`), which is
   the combination that triggers the bug:

```matlab
% ── create_wind_source_selector.m ──────────────────────────────────────────
repro_dir = fullfile(tempdir, 'when_t_repro');
mkdir(repro_dir);
cd(repro_dir);

mdl = 'WindSourceSelector';
sfnew(mdl);
set_param(mdl, 'SolverType', 'Variable-step', 'Solver', 'VariableStepAuto', 'StopTime', '10.0');
set_param([mdl '/Chart'], 'Name', 'Selector');

rt = sfroot();
m  = rt.find('-isa', 'Simulink.BlockDiagram', 'Name', mdl);
ch = m.find('-isa', 'Stateflow.Chart');
ch.ChartUpdate = 'INHERITED';

% Inputs
function d = inp(ch, name, dtype)
    d = Stateflow.Data(ch); d.Name = name; d.Scope = 'Input'; d.DataType = dtype;
end
inp(ch, 'BAnem1Fault',  'boolean');
inp(ch, 'BAnem2Fault',  'boolean');
inp(ch, 'NAnemSpeed1',  'single');
inp(ch, 'NAnemSpeed2',  'single');
inp(ch, 'NSafeSpeed',   'single');
inp(ch, 'NMaxFaults',   'uint16');
inp(ch, 'BReset',       'boolean');

% Outputs (NActiveAnem is uint16 — this mixed type is required to trigger the bug)
function d = outp(ch, name, dtype, init)
    d = Stateflow.Data(ch); d.Name = name; d.Scope = 'Output';
    d.DataType = dtype; d.Props.InitialValue = init;
end
outp(ch, 'NSpeedOut',    'single',  '0');
outp(ch, 'NActiveAnem',  'uint16',  '1');
outp(ch, 'BBothFaulted', 'boolean', 'false');
outp(ch, 'NAnem1FltCnt', 'single',  '0');
outp(ch, 'NAnem2FltCnt', 'single',  '0');

% States
sA1   = Stateflow.State(ch); sA1.Position   = [40 40  200 90];
sA1.LabelString = sprintf(['UsingAnem1\n' ...
    'en: NActiveAnem=uint16(1);\n' ...
    'du: if BAnem1Fault, NAnem1FltCnt=NAnem1FltCnt+1; else NAnem1FltCnt=0; end\n' ...
    '    if BAnem2Fault, NAnem2FltCnt=NAnem2FltCnt+1; else NAnem2FltCnt=0; end\n' ...
    '    NSpeedOut=NAnemSpeed1; BBothFaulted=false;']);

sA2   = Stateflow.State(ch); sA2.Position   = [40 160 200 80];
sA2.LabelString = sprintf(['UsingAnem2\n' ...
    'en: NActiveAnem=uint16(2);\n' ...
    'du: if BAnem2Fault, NAnem2FltCnt=NAnem2FltCnt+1; else NAnem2FltCnt=0; end\n' ...
    '    NSpeedOut=NAnemSpeed2; BBothFaulted=false;']);

sBoth = Stateflow.State(ch); sBoth.Position = [40 270 200 60];
sBoth.LabelString = sprintf(['BothFaulted\n' ...
    'en: NActiveAnem=uint16(0); BBothFaulted=true;\n' ...
    'du: NSpeedOut=NSafeSpeed;']);

% Junctions
j1 = Stateflow.Junction(ch); j1.Position.Center = [320  90];
j2 = Stateflow.Junction(ch); j2.Position.Center = [320 200];
j3 = Stateflow.Junction(ch); j3.Position.Center = [320 300];

% Default transition
td = Stateflow.Transition(ch); td.Destination = sA1; td.DestinationOClock = 9;

% Fault transitions
t1j1 = Stateflow.Transition(ch); t1j1.Source = sA1;  t1j1.Destination = j1;
t1j1.LabelString = '[NAnem1FltCnt>=NMaxFaults]';
tj1a2 = Stateflow.Transition(ch); tj1a2.Source = j1; tj1a2.Destination = sA2;
tj1a2.LabelString = '[~BAnem2Fault]';
tj1bf = Stateflow.Transition(ch); tj1bf.Source = j1; tj1bf.Destination = sBoth;
t2bf  = Stateflow.Transition(ch); t2bf.Source  = sA2; t2bf.Destination  = sBoth;
t2bf.LabelString = '[NAnem2FltCnt>=NMaxFaults]';

% Reset transitions
tr1 = Stateflow.Transition(ch); tr1.Source = sA2;   tr1.Destination = j2; tr1.LabelString = '[BReset]';
tj2 = Stateflow.Transition(ch); tj2.Source = j2;    tj2.Destination = sA1;
tr2 = Stateflow.Transition(ch); tr2.Source = sBoth; tr2.Destination = j3; tr2.LabelString = '[BReset]';
tj3 = Stateflow.Transition(ch); tj3.Source = j3;    tj3.Destination = sA1;

% Save & reload, then wire Inport/Outport blocks
save_system(mdl, fullfile(repro_dir, [mdl '.slx'])); close_system(mdl, 0);
load_system(fullfile(repro_dir, [mdl '.slx']));

chartPath = [mdl '/Selector'];
sfPH = get_param(chartPath, 'PortHandles');

inNames  = {'BAnem1Fault','BAnem2Fault','NAnemSpeed1','NAnemSpeed2','NSafeSpeed','NMaxFaults','BReset'};
outNames = {'NSpeedOut','NActiveAnem','BBothFaulted','NAnem1FltCnt','NAnem2FltCnt'};

for i = 1:numel(inNames)
    blk = [mdl '/' inNames{i}];
    add_block('simulink/Sources/In1', blk, 'Position', [20 30+60*(i-1) 50 60+60*(i-1)]);
    ph = get_param(blk, 'PortHandles');
    add_line(mdl, ph.Outport(1), sfPH.Inport(i));
end
for i = 1:numel(outNames)
    blk = [mdl '/' outNames{i}];
    add_block('simulink/Sinks/Out1', blk, 'Position', [500 30+60*(i-1) 530 60+60*(i-1)]);
    ph = get_param(blk, 'PortHandles');
    add_line(mdl, sfPH.Outport(i), ph.Inport(1));
end

save_system(mdl); close_system(mdl, 0);
fprintf('Created: %s\n', fullfile(repro_dir, [mdl '.slx']));
```

2. Create `wind_repro.feature` in the same directory:

```gherkin
# --- front-matter:toml ---
model = "WindSourceSelector.slx"
[inputs]
BAnem1Fault  = "BAnem1Fault"
BAnem2Fault  = "BAnem2Fault"
NAnemSpeed1  = "NAnemSpeed1"
NAnemSpeed2  = "NAnemSpeed2"
NSafeSpeed   = "NSafeSpeed"
NMaxFaults   = "NMaxFaults"
BReset       = "BReset"
[outputs]
NActiveAnem = "NActiveAnem"
# --- end front-matter ---

Feature: when-t time format comparison in draft mode

Scenario: integer seconds - draft mode
  Given inputs
    * BAnem1Fault  = const(0)
    * BAnem2Fault  = const(0)
    * NAnemSpeed1  = const(10)
    * NAnemSpeed2  = const(12)
    * NSafeSpeed   = const(0)
    * NMaxFaults   = const(5)
    * BReset       = const(0)
  When simulate for 3s in Normal mode
  Then outputs
    * ActiveAfterInit: NActiveAnem == 1 when t > 1s

Scenario: decimal seconds - draft mode
  Given inputs
    * BAnem1Fault  = const(0)
    * BAnem2Fault  = const(0)
    * NAnemSpeed1  = const(10)
    * NAnemSpeed2  = const(12)
    * NSafeSpeed   = const(0)
    * NMaxFaults   = const(5)
    * BReset       = const(0)
  When simulate for 3s in Normal mode
  Then outputs
    * ActiveAfterInit: NActiveAnem == 1 when t > 0.5s

Scenario: milliseconds - draft mode
  Given inputs
    * BAnem1Fault  = const(0)
    * BAnem2Fault  = const(0)
    * NAnemSpeed1  = const(10)
    * NAnemSpeed2  = const(12)
    * NSafeSpeed   = const(0)
    * NMaxFaults   = const(5)
    * BReset       = const(0)
  When simulate for 3s in Normal mode
  Then outputs
    * ActiveAfterInit: NActiveAnem == 1 when t > 500ms
```

3. Run in MATLAB:

```matlab
% Fails — all 3 scenarios
model_test('WindSourceSelector.slx', 'wind_repro.feature', '[]', 'false', 'true')

% Passes — all 3 scenarios
model_test('WindSourceSelector.slx', 'wind_repro.feature', '[]', 'false', 'false')
```

4. **Observe:** With `draft_mode='true'` (last argument), all 3 scenarios fail. With
   `draft_mode='false'`, all 3 pass.

### Minimal Reproduction

```matlab
% Single scenario, integer seconds (simplest possible case) — still fails in draft_mode='true'
model_test('WindSourceSelector.slx', 'wind_repro.feature', ...
    '["integer seconds - draft mode"]', 'false', 'true')
```

## Environment

### Toolkit & Agent

| Detail | Value |
|--------|-------|
| **SATK Version** | commit `cfcf2568` — development build (`2026.04.17`) |
| **MCP Server Version** | v0.8.1 |
| **Agent / Client** | GitHub Copilot CLI |
| **MCP Server Mode** | attach-to-existing (shared MATLAB session) |
| **Loaded Skills** | `testing-simulink-models`, `building-simulink-models`, `filing-bug-reports` |

### MATLAB

| Detail | Value |
|--------|-------|
| **MATLAB Version** | 24.2.0.3070828 (R2024b) Update 7 |
| **Simulink Version** | 24.2 |
| **Stateflow Version** | 24.2 |
| **Simulink Test** | 24.2 (R2024b) |

### Platform

| Detail | Value |
|--------|-------|
| **OS** | Windows 11 Enterprise Build 26200 |
| **Architecture** | win64 |

## Error Output

```
Warning: Resetting model WindSourceSelector to clear stale harness associations.
> In ModelTestRun.cache.Cache.ensureHarness
In ModelTestRun.createSimulinkTestFile
In model_test

================== test session starts ==================
model: WindSourceSelector.slx
gherkin: ...\wind_repro.feature
scenarios: 3

FAILED integer seconds - draft mode (12.44s)
  FAILED: Unable to evaluate assessments because of errors during test
          case setup or model simulation (see test case results section
          for more information)
FAILED decimal seconds - draft mode (1.05s)
  FAILED: Unable to evaluate assessments because of errors during test
          case setup or model simulation
FAILED milliseconds - draft mode (0.82s)
  FAILED: Unable to evaluate assessments because of errors during test
          case setup or model simulation

=============== 0 passed, 3 failed in 14.31s =============
                 (0 passed, 3 failed, 0 untested of 3 assertions)
```

Compare with `draft_mode='false'` (all pass):

```
================== test session starts ==================
scenarios: 3
PASSED integer seconds - draft mode (1.93s)
PASSED decimal seconds - draft mode (1.34s)
PASSED milliseconds - draft mode (0.87s)
=================== 3 passed in 4.14s ===================
                 (3 passed, 0 failed, 0 untested of 3 assertions)
```

## Impact

- **Scope:** All users of `model_test` with `draft_mode='true'` and any `when t > X` conditional
  assessment.
- **Workaround:** Use `draft_mode='false'`. This increases compile time (~60s vs ~3s per scenario)
  but produces correct results.
- **Blocking:** `draft_mode='true'` is documented as the recommended default for rapid iteration.
  This effectively disables the fast-iteration workflow for any test using conditional time-windowed
  assertions.

## Related Files

| File | Relevance |
|------|-----------|
| `tools/model_test/+ModelTestRun/Assessment.p` | Assessment evaluation logic for `when t` clauses |
| `tools/model_test/+ModelTestRun/createSimulinkTestFile.p` | Creates Simulink Test harness; draft vs normal mode paths diverge here |
| `tools/model_test/+ModelTestRun/ThenOutputsBullet.p` | Parses `Then outputs` bullets including `when t` syntax |
| `skills-catalog/.../testing-simulink-models/SKILL.md` | Skill docs — contains a note about `when t`; should be updated when this is fixed |

## Notes

*Speculation — for investigator:*

- The first `draft_mode=true` run takes ~12s (similar to a full compile), but subsequent runs take
  ~1s, suggesting the harness is built on first run, yet assessment evaluation still fails.
- The warning "Resetting model to clear stale harness associations" appears consistently before the
  failure; may indicate harness state management differs in draft mode.
- **Mixed Stateflow data types are required to trigger the bug.** A Stateflow chart using all
  `double` ports passes both modes correctly. The bug only reproduces when the chart has mixed types
  (`boolean`, `single`, `uint16`). In the reproduction model, `NActiveAnem` is `uint16` — this is
  likely the type that confuses draft mode's assessment evaluation when paired with a `when t`
  clause.
- Junctions in the Stateflow chart also appear necessary; a structurally simpler chart (no
  junctions) with the same data types has not yet been isolated as a minimal trigger.


### Additional Context

_No response_

Detail	Value
SATK Version	commit `cfcf2568` — development build (`2026.04.17`)
MCP Server Version	v0.8.1
Agent / Client	GitHub Copilot CLI
MCP Server Mode	attach-to-existing (shared MATLAB session)
Loaded Skills	`testing-simulink-models`, `building-simulink-models`, `filing-bug-reports`

File	Relevance
`tools/model_test/+ModelTestRun/Assessment.p`	Assessment evaluation logic for `when t` clauses
`tools/model_test/+ModelTestRun/createSimulinkTestFile.p`	Creates Simulink Test harness; draft vs normal mode paths diverge here
`tools/model_test/+ModelTestRun/ThenOutputsBullet.p`	Parses `Then outputs` bullets including `when t` syntax
`skills-catalog/.../testing-simulink-models/SKILL.md`	Skill docs — contains a note about `when t`; should be updated when this is fixed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true` #8

Bug Report

BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true`

Summary

Expected Behavior

Actual Behavior

Reproduction Steps

Minimal Reproduction

Environment

Toolkit & Agent

MATLAB

Platform

Error Output

Impact

Related Files

Notes

Additional Context

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Detail	Value
MATLAB Version	24.2.0.3070828 (R2024b) Update 7
Simulink Version	24.2
Stateflow Version	24.2
Simulink Test	24.2 (R2024b)

Uh oh!

BUG-001: [model_test] — when t conditional assessments — All fail silently in draft_mode=true #8

Description

Bug Report

BUG-001: [model_test] — when t conditional assessments — All fail silently in draft_mode=true

Summary

Expected Behavior

Actual Behavior

Reproduction Steps

Minimal Reproduction

Environment

Toolkit & Agent

MATLAB

Platform

Error Output

Impact

Related Files

Notes

Additional Context

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true` #8

BUG-001: [model_test] — `when t` conditional assessments — All fail silently in `draft_mode=true`