Verifying And Validating Simulation Models Topic 03
Verifying And Validating
Simulation Models
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 1
Verifying And Validating Simulation Models Topic 03
Verification And Validation References
VERIFICATION AND VALIDATION OF SIMULATION MODELS, Robert G. Sargent, Proceedings
of the 2011 Winter Simulation Conference, S. Jain, R.R. Creasey, J. Himmelspach, K.P. White,
and M. Fu, eds, http://www.informs-sim.org/wsc11papers/016.pdf
Discrete-Event System Simulation, Banks, Carson, Nelson, Nicol, Fifth Edition, Prentice Hall.
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 2
Verifying And Validating Simulation Models Topic 03
Verifying And Validating Simulation Models
To be useful, simulation models need to:
Work correctly for the intended purpose
Be acceptable to users who intend to use it to make decisions
The processes of ensuring that simulation models are useful are often referred to as:
Verification
Validation
Use of models that are not both verified and validated can lead to:
Incorrect and possibly dangerous and/or expensive results
Lack of acceptance by users who need it to make decisions
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 3
Verifying And Validating Simulation Models Topic 03
Verification
Sargent defines verification as the process of ensuring that the computer program of the
computerized model and its implementation are correct
In other words, verification is the process of ensuring that the model you build works the way
you intended it to work
Nonetheless, verified models may be totally incorrect in that the way you intended the model to
work may be incorrect
Sargent, R. G. 2000. Verification, Validation, and Accreditation of Simulation Models. In Proc.
2011 Winter Simulation Conf.
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 4
Verifying And Validating Simulation Models Topic 03
Validation
Schlesinger defines validation as substantiation that a computerized model within its domain
of applicability possesses a satisfactory range of accuracy consistent with the intended
application of the model)
Note that a model validated for one purpose may not be valid for another
Schlesinger, S., R. E. Crosbie, R. E. Gagn, G. S. Innis, C. S. Lalwani, J. Loch, R. J. Sylvester, R.
D. Wright, N. Kheir, and D. Bartos. 1979. Terminology for Model Credibility. Simulation
32(3):103-104.
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 5
Verifying And Validating Simulation Models Topic 03
Model Building Steps - Step 1 - Observation
Banks suggests that the first step in model building consists of observing the real system and
the interactions among its various components and of collecting data on their behavior.
Banks further suggests that you may need to repeat this step at various times in the model
building process
Banks, Carson, Nelson, Nicol, Discrete-Event System Simulation, Prentice Hall.
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 6
Verifying And Validating Simulation Models Topic 03
Model Building Steps - Step 2 - Construction Of Conceptual Model
Banks suggests that the second step in model building is the construction of a conceptual
model a collection of assumptions about the components and the structure of the system,
plus hypotheses about the values of model input parameters
To validate the conceptual model (part of the whole validation process) it is important to keep a
record of these assumptions and hypotheses
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 7
Verifying And Validating Simulation Models Topic 03
Model Building Steps - Step 3 Model Implementation
Banks further suggests that the third step is to implement the conceptual model using
appropriate simulation software
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 8
Verifying And Validating Simulation Models Topic 03
Verifying Simulation Models
Before replicating the model:
Document the specific tests you will perform in advance of performing them
Create range names for parameters and initial values
Check all input parameters for correctness
Check all formulas
Add additional formulas with intermediate values to ensure those values are correct
Calculate additional summary statistics so as to ensure they are correct
Force the model to possibly unusual conditions by adjusting range values and make sure that you
get desired results
Present the model for review to other simulation modelers; note that just explaining to another
individuals how the simulation works can help identify errors
After replicating the model repeat the above tests to determine the effect of the tests on
summary statistics
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 9
Verifying And Validating Simulation Models Topic 03
Verifying Other Simulation Models
Additional steps (not appropriate for spreadsheet based simulation models) for verifying
simulation models developed using simulation software:
Visually animate the model
Visually review the model
Visually review all simulation logic
Start up the simulation monitor and step through the simulation:
Watch as many variables as possible
With just one work item in it at a time and make sure it flows properly
With a few more work items and make sure the simulation still flows properly
Check time and content of future events in the monitor
Step through any visual logic you have
Run the simulation with a variety of combinations of parameters and check the results of each
combination for reasonableness
Present all of the above one at a time to other modelers
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 10
Verifying And Validating Simulation Models Topic 03
Validation Of Assumptions
For statistical assumptions
Make sure the data you obtain is accurate and representative as inaccurate data is often the norm
because:
Data was retrieved from the wrong field or table
Data was never recorded properly in the first place
Data may have been rounded inappropriately
Data may have been under or over exaggerated to make individuals or departments look better (or worse)
Data may be recorded with different units (Celsius, Fahrenheit, pounds, kilograms, . . .)
The sample of time during which the data was collected may not be representative of what is being analyzed
Apply standard statistical hypothesis tests
For model process or logic assumptions, review assumptions with stakeholders and Subject
Matter Experts (SMEs)
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 11
Verifying And Validating Simulation Models Topic 03
Validation Against Historical data
With the help of stakeholders and SMEs, build a test data set (in a data input file) for which you
know the results; that test data set is generally based on historical inputs
Use that test data set to test the model
Compare the outputs with the test data set to reality; if differences are not explainable then you
most likely have a problem you need to address
Run the test data set for which you know the results whenever you make changes to the
simulation model
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 12
Verifying And Validating Simulation Models Topic 03
Stakeholder Validation
A simulation model will not be very useful if stakeholders are not convinced that it represents
their process
To obtain that validation:
Inform stakeholders in advance of need for their validation of model
Get input from stakeholders and SMEs as to important details of the model (throughout the modeling
process)
Log assumptions
Track data sources
Animate simulation model
Run model with real historical data and compare it to actual results
Have review meetings that includes all of the above
Respond to stakeholder and SME feedback
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 13
Verifying And Validating Simulation Models Topic 03
Graded Homework Questions
After watching the video and reading Robert Sargent's article (see list of references at beginning
of these slides), in your own words, briefly answer the following questions:
Why is verification and validation important?
What is IV&V and what are the benefits of using it as opposed to other approaches for verifying and
validating models?
What is the difference between conceptual model validation, computerized model verification, and
operational validation?
List and very briefly explain at least six different validation techniques
Briefly explain what a Turing Test is (see en.wikipedia.org/wiki/Turing_test ) and then how Turing
Tests be used to validate a simulation model
Briefly explain why documenting verification and validation activities is important
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 14
Verifying And Validating Simulation Models Topic 03
Graded Homework Problems
Develop a written verification plan for your implementation of Topic 2 Problem 12
Philip Troy, 2008 - 2015 [email protected] www.PhilTroy.com 15
