FORECASTING AND MODEL SELECTION - IIT Kanpur and Forecasting Forecasting Methods Measuring the Accuracy of Forecast FORECASTING AND MODEL SELECTION Anurag Prasad Department of Mathematics

Modeling and ForecastingForecasting Methods

Measuring the Accuracy of Forecast

FORECASTING AND MODEL SELECTION

Anurag Prasad

Department of Mathematics and StatisticsIndian Institute of Technology Kanpur, India

REACH Symposium, March 15-18, 2008

1 Forecasting and Model Selection



Outline

1 Modeling and Forecasting

2 Forecasting Methods

3 Measuring the Accuracy of Forecast




Assumptions of Forecasting

1 Element of Uncertainty

2 Blind Spots

3 Change in Forecast Accuracy




Framework of a Forecast System

No

Historical Data

Data

SpecificationModel

Checking

Diagnostic Estimation

Model ForecastGeneration

StabilityChecking

ForecastUpdation

NewObservations

Yes Yes

Model−Building Phase Forecasting Phase

No

Theory and/or




Choice of a Particular Forecast Model

1 Degree of Accuracy Required

2 Cost of Producing Forecasts

3 Forecast Horizon

4 Degree of Complexity Required

5 Available Data




Classification of Estimation Methods

1 Time Series Methods

2 Causal Methods

3 Judgemental Methods




Time Series Methods

• Use historical data as a basis• Underlying patterns are fairly stable

1 Autoregressive Moving Average (ARMA)2 Exponential Smoothing3 Extrapolation4 Linear Prediction5 Trend Estimation6 Growth Curve7 Box-Jenkins Approach




Causal Methods

• Belief that some other time series can be useful• Assumption that it is possible to identify the underlying factors

1 Regression Analysis* Linear Regression* Non-Linear Regression

2 Econometrics




Judgemental Methods

• Incorporate intuitive judgements, opinions and probabilityestimates

1 Composite Forecasts2 Surveys3 Delphi Method4 Scenario Building5 Technology Forecasting6 Forecast by Analogy




Forecast Error

For t = 1, . . . , N,y(t) : Actual value at period t ,y(t) : Forecast value at period t ;e(t) : Forecast error at period t ; e(t) = y(t) − y(t)

i

y(t )i

t it

y(t)

y(t )

^




Graphical Measures of Forecast Accuracy

Plot of y(t) versus y(t)

• Keep the same scale for both the axes.• Departure of points from the 450 line through origin indicatesimperfect forecasts.




Correct Model Form

y(t)

y(t)Incorrect Model Form

y(t)

y(t)




Plot of e(t) versus t

• Reveals patterns of variability which the model has failed toexplain.• For a good model, the forecast errors should vary in ahorizontal band around zero.




0

tCorrect Model Form

0

tIncorrect Model Form

0


0

e(t) e(t)

e(t) e(t)





Descriptive Measures of Forecast Accuracy

Descriptive Measures of Forecast Accuracy are used to...

1 Provide a single, easily interpreted measure of model’sreliability

2 Compare the accuracy of two different models3 Search for an optimal model4 Monitor a model’s performance





Mean Absolute Deviation (MAD)

MAD =�

|forecast error|number of forecasts =

� Nt=1 |e(t)|

N

Mean Square Error (MSE)

MSE =�

(forecast error)2

number of forecasts =� N

t=1 e(t)2

N

Root Mean Square Error (RMSE)

RMSE =√

MSE





Mean Absolute Percent Error (MAPE)

MAPE =�

|forecast error/actual value|number of forecasts .100%

=� N

t=1 |e(t)/y(t)|N .100%

Pearson’s Correlation Coefficient (r )between y(t) and y(t)

r =� N

t=1(y(t)−y)( �y(t)−¯�y)� � Nt=1(y(t)−y)2

� � Nt=1( �y(t)−¯�y)2





"No Change" model is : y(t + 1) = y(t)

Theil’s Inequality Coefficient (U)

U = RMSE("new" model)RMSE("no change" model)

U > 1 ⇒ worse than "no change" model

U = 1 ⇒ as good as "no change" model

U < 1 ⇒ better than "no change" model




Further Readings

Quantitative Forecasting Methods, N.R. Farnum and L.W.Stanton, 1989, PWS-KENT Publishing Co.

Statistical Methods for Forecasting, B. Abraham and J.Ledolter, 1983, John Wiley & Sons

Introduction to Time Series and Forecasting, P.J. Brockwelland R.A. Davis, 2002 (Second Edition) , Springer-Verlag

Time Series Analysis and Forecasting, R. Yaffee, 2000,Academic Press




Thank You


Documents

FORECASTING AND MODEL SELECTION - IIT Kanpur and Forecasting Forecasting Methods Measuring the Accuracy of Forecast FORECASTING AND MODEL SELECTION Anurag Prasad Department of Mathematics