Lecture 7(a): Ensuring
“Correctness” in Simulation
1
Outline
Verification
Validation
Credibility
2
Readings
Law – Chapter 5
Sargent, "Verification and Validation of
Simulation Models” (paper posted on
Moodle)
3
Verification
System description and assumptions
Is the
translation
correct?
Simulation program
4
Verification Techniques
Technique 1: start from smaller/simpler
models or sub-models
Why?
Example: for a multi-teller bank with balking,
write a simulation without balking first
Balking occurs when potential customers arriving at a queueing system
choose not to enter it.
5
Verification Techniques
Technique 2: second opinion
Why?
Example: divide your team into sub-teams
working on different parts of the model, then
give to each other to determine and/or fix bugs
6
Verification Techniques
Technique 3: test the model under a
variety of conditions and check if the
output is reasonable
Why?
Example: check that utilization is reasonable
under a variety of arrival and processing rates
7
Verification Techniques Why?
Technique 4: trace – display the values
of variables, attributes, counters after
every event Arena: step
Compare to hand calculations
Example: check that utilization is reasonable
under a variety of arrival and processing rates
8
Verification Techniques Why?
Technique 5: run the model under
simplifying assumptions for which its
characteristics are known or easily
computed
Example: for a job shop model with multiple
stations, run a simpler model with one station
(for each station) and compare to analytical
results from queueing theory
9
Verification Techniques Why?
Technique 6: observe animation of the
simulation output
Example: queue build-up at a station, more
entities going on a particular branch of the
decide module
10
Verification Techniques Why?
Technique 7: double-check your input
(distributions, parameters, time units)
Example: Expo(mean) – did you use the mean
or the rate?
rate
11
Validation
Objectives of the study
Accurate
representation
for the
objectives?
Simulation program
12
Validation Techniques
Validation Technique 1: comparison
with existing system
Sample statistics
Confidence intervals
Formal tests
Compare current state performance with
current actual performance! (Q: what
performance metrics would you use?)
13
Validation Techniques
Validation Technique 2: comparison
with expert opinion
Stakeholders validate that the output of
the model is representative of reality
Stakeholders cannot tell difference
between output from model and reality
(i.e., model should produce output as
“good” as the real system)
14
Validation Techniques
Validation Technique 3: comparison
with another model
If the other model exists and has already
been validated 😏
15
Validation Techniques
Validation Technique 4: comparison via
animation
Determine if the system behaves “as
expected” via animation
E.g., are all potential paths of the entities
represented? Are the queues building up
where we expected?
16
Validation Techniques
Validation Technique 1: numerical
comparison with existing system
Sample statistics
Confidence intervals
Formal tests
Focus on this one if possible!
17
Establishing Credibility
Stakeholders
Do
stakeholders
accept the
model?
Simulation program
18
Techniques for Establishing
Credibility
Technique 1: Maintain contact with the
stakeholders (show them the model,
with animation!)
Technique 2: Maintain a list of
assumptions/specifications (refer to it
while interacting with the stakeholders!)
19
Questions?
20
