1

Model Verification and

Validation

2 2

Outline

Important of Model Verification and

Validation

Model Verification

Model Validation

Model Building, Verification and Validation

3

Important of Model Verification and

Validation

Model building is, by nature, very error prone

The modeler must translate the real-world system into a conceptual model

Then the conceptual model must be translated into a simulation models.

This translation is iterative

In this, there is plenty of room for making errors

Verification and validation processes can reduce or eliminate these errors.

4

Important of Model Verification and

Validation

Model Verification is the process of determining whether the simulation model correctly reflects the conceptual model.

Model Validation is the process of determining whether the conceptual model correctly reflects the real system

Model Verification and Validation are critical to the success of simulation project.

5

6

Reason for Neglect

The primary reasons for neglecting this

important activity are:

Time and budget pressures

Laziness

Overconfidence

Ignorance

7

Practices That Facilitate Verification

and Validation

Poor modeling practices

Build models with little or no thought about being able to verify or validate the model

Models contain spaghetti code that is difficult for anyone, including the modeler, to follow.

Becomes more acute as models grows in complexity

8

Practices That Facilitate Verification

and Validation

To create models that ease the difficulty of

Verification and Validation

Reduce the amount of complexity of the model

The code is readable and understandable

Finally, model data and logic code should be

thoroughly and clearly documented

9 3

Objectives of Verification and

Validation

To produce a representative model of the

system under study

To increase the model credibility

To gradually refine the model during the

development process

10

Model Verification

Model Verification is the process of determining whether the model operates as intended (it runs correctly).

Building the model right

Tries to detect unintended errors in the model data and logic and removes them.

Verification is the process of debugging the model

11

Model Verification

Errors or bugs in a simulation model are of

two types:

Syntax errors grammatical errors

Semantic errors

Associated with the meaning of the modeler and are therefore difficult to detect.

Often they are logical errors

12

Model Verification

Preventive Measures

Get it right the first time (models with no bugs the first time)

In practice, this isnt always possible as bugs are often sneaky and difficult to prevent

Practices minimizing bugs are:

BE CAREFUL

13

Preventive Measures

Use structured programming

Five principles of structured programming

Top-down design

Modularity

Compact modules

Stepwise refinement

Structured control

14

Establishing a Standard for

Comparison

One simple standard is a common sense

If your simulation is giving totally bizarre results

To construct an analytic model of the problem

(with simplified assumption) if at all possible,

e.g.,

To run simulation without downtime

A queuing system without balking

15

Verification Techniques

Conduct model code reviews

Check the output for reasonableness

Watch simulation for correct behavior

Use the trace and debug facilities provided

with the software

16

Model Validation

Model Validation is the process of determining whether the model is meaningful and accurate representation of the real system (Hoover and Perry, 1990)

Building the right model

Stakeholders and customers should become heavily involved in the validation process

Can be very time consuming

17

Model Validation

Not interested in achieving absolute validity

but only functional validity.

Functional validity: The process of

establishing that the models output behavior has sufficient accuracy for the models intended purpose over the domain of the

models intended applicability (Sargent, 1998)

18

Model Validation

Through simulation we convey the user or consumer that the simulation results can be trusted and used to make a real world decisions.

For existing systems, the model behavior should correspond to that of the actual system.

In case of a new system, the input data should accurately reflect the design specification of the system.

19

Model Validation

Determining Model Validity

There is no simple test to determine the

validity of a model.

Validation is an inductive process

As with model verification, it is common to

use a combination of techniques when

validating a model.

20

Model Validation

Techniques for validating a model

Watching the animation

Comparing with other models

Conducting degeneracy and extreme condition test

Checking for face validity

Testing against historical data

Performing sensitivity analysis

Running traces

Conducting Turing test

21 9

Steps in Validation

1.- Build a model with high face validity.

2.- Validate model assumptions.

3.- Compare the models input-output transformations against those in the real

system.

22 10

Face Validity

Face validity is concerned with the

reasonableness of the model to knowledgeable

peers.

Sensitivity analysis can help checking for face

validity.

23 11

Validating Model Assumptions

Types of Assumptions

Structural

Data

Structural assumptions must be checked

against the real system.

Data assumptions must be checked by

statistical testing.

24 12

Validating Transformations

The range of outputs of the model for a given

range of inputs must resemble the one

observed in the real system.

Use historical data.

Validate on the main response variables.

What to do if the model represents a non-

existing system?

25

Maintaining Validation

We need to maintain validation as system

specifications tend to evolve right up to, and

often even after system implementation

We need to maintain validation (keep

updating the model to continually reflect

current system design specification)

26

Optimum Level of Validation

27 5

Model Building, Verification and

Validation

Steps in Model Building

1.- Observe the real system

2.- Construct conceptual model and perform

conceptual validation

3.- Translate conceptual model into a computer

model and perform verification

4.- Calibrate, verify and validate at every step

28 4

Verification and Validation

Verification

Building the model right

Validation

Building the right model

Verification and Validation must be conducted

simultaneously throughout the model

development process

29 8

Validation and Calibration

Validation compares the model to the real

system.

Calibration adjusts the model to make it more

representative of the real system.

Validation and Calibration must be performed

all the time and until the very last minute.

30

Statistical Techniques for Validation

Analysis of variance

Multiple analysis of variance

Confidence intervals

Goodness-of-fit tests (Chi-square test, Kolmogorov-Smirnov test)

Tests of means (t-test, nonparametric)

Regression analysis

Time-series analysis