Skip to content

Add scenario-aware exploration to motel check#143

Merged
andrewh merged 1 commit into
mainfrom
claude/check-scenario-aware
Jun 11, 2026
Merged

Add scenario-aware exploration to motel check#143
andrewh merged 1 commit into
mainfrom
claude/check-scenario-aware

Conversation

@andrewh

@andrewh andrewh commented Jun 10, 2026

Copy link
Copy Markdown
Owner

Closes #70 (part of the property-testing epic, issue 62).

motel check previously analysed the topology in isolation: a topology that passes all checks could still violate bounds when a scenario adds calls or retries. It now reads the scenarios: block already present in the topology YAML — per the design discussion on the issue, no new input flags are needed.

How it works

  • Combination enumeration: scenario windows are swept at their start/end boundaries to find every distinct combination of co-active scenarios (active sets only change at boundaries, so this covers every combination that occurs in time). This answers the issue's open questions: rather than varying activation randomly per sample or sweeping continuously, every reachable combination is checked deterministically.
  • Static analysis under overrides: worst-case depth, fan-out, and span analysis apply each combination's add_calls/remove_calls via the engine's existing effectiveCalls resolution. Exported MaxDepth/MaxFanOut/MaxSpans/SampleTraces signatures are unchanged; they delegate to override-aware internals.
  • Sampled exploration under overrides: each combination is sampled with its overrides active, using a shared seed so combinations sample the same trace sequence.
  • Worst-case attribution: each check reports the combination that produced the worst case (by static value, sampled maximum as tie-breaker; ties go to the baseline) in a new CheckResult.Scenarios field, rendered as a scenarios: annotation in the CLI output.
  • Opt-out: --skip-scenarios restores baseline-only checking, per the issue discussion.

Example

$ motel check topology.yaml --samples 100
PASS  max-depth: 1 (limit: 10)
      path: gateway.handle → backend.process
      p50: 1  p95: 1  p99: 1  max: 1  (100 samples)
PASS  max-fan-out: 5 (limit: 100)
      worst: gateway.handle
      scenarios: retry storm
      p50: 2  p95: 2  p99: 2  max: 2  (100 samples)
PASS  max-spans: 6 static worst-case, 3 observed/100 samples (limit: 10000)
      scenarios: retry storm
      p50: 3  p95: 3  p99: 3  max: 3  (100 samples)

Testing

  • Unit tests for ScenarioSets enumeration (empty, overlapping windows, disjoint dedup) and for Check attribution (scenario producing the worst case fails a tight limit and is named; baseline wins ties and call-removing scenarios)
  • CLI tests covering the scenarios: output annotation and --skip-scenarios
  • All existing check/property tests pass unchanged; make test and make lint green

Independent of PR 140 — branched from main.

https://claude.ai/code/session_01TgWavXRfzQiTWDZccuS2pf


Generated by Claude Code

motel check now reads the scenarios block from the topology and checks
every distinct combination of co-active scenarios alongside the
baseline. Scenario windows are swept at their boundaries to enumerate
combinations; each combination's call overrides (add_calls,
remove_calls) are applied to both static analysis and sampled
exploration. Each check reports the worst case and names the scenario
combination that produced it, selected by static value with the sampled
maximum as tie-breaker. A --skip-scenarios flag restores baseline-only
checking.

Closes #70
@andrewh
andrewh merged commit 233f644 into main Jun 11, 2026
2 checks passed
@andrewh
andrewh deleted the claude/check-scenario-aware branch June 11, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario-aware exploration in motel check

2 participants