May 29, 2012
DETM/DIRI
The Economics and Econometrics of
Commodity Prices – 2012
João Victor Issler (FGV) and Claudia F. Rodrigues (VALE)
August 17, 2012
Comparing Forecast Accuracy
of Different Models for Prices
of Metal Commodities
Forecasting within a panel data framework
■ There are many ways to combine forecasts:
1. bias-corrected average forecast
2. simple average forecast
3. weighted average forecast based on MSE
■ Our aim:
evaluate these forecast combinations according to their predictability performance measured by the RMSE.
compare the predictability of: forecast combinations x “best-model” forecast (BIC).
■ According to the errors decomposition formula:
fhi,t = yt + κi + εi,t + ηt
■ i indicates a model; i = 1,..., N
■ fhi,t stands for forecasts of yt computed using conditioning sets
lagged h periods
■ κi time-invariant bias
■ εi,t idiosyncratic risk
■ ηt aggregate shock
Analyzing the forecast errors
ηt
κi Bias Idiosyncratic shock
εi,t
Analyzing the forecast errors
Aggregate shock
ηt
κi Bias Idiosyncratic shock
εi,t
Analyzing the forecast errors
Aggregate shock
Shocks that affect all forecasts
Errors of idiosyncratic nature Model misspecification or unknown risk function
ηt
κi Bias Idiosyncratic shock
εi,t
Analyzing the forecast errors
Aggregate shock
Shocks that affect all forecasts
Model misspecification or unknown risk function
Errors of idiosyncratic nature
Combine forecasts Estimate the bias
Cannot avoid it
The series
■ Target variable
■ the metals price index¹ (IMF)
■ Co-variates used in forecasting
■ US industrial production
■ Chinese industrial production
■ Primary metals coincident index (USGS²)
■ Leading index of metals price (USGS²)
■ VIX – a volatility index
■ US real effective exchange rate
■ S&P500
¹ an aggregation of weighted average of individual indices: copper, aluminum, iron ore, nickel, tin, zinc, lead and uranium. ² US Geological survey
The series
■ Correlations between the target series and each covariate in dlog:
US IP China IP Coincident
index
Leading
index VIX REER S&P500
Lag 1 0.13 0.26 0.16 0.25 -0.17 -0.15 0.11
■ Time span
■ 01/01/1980 to 01/05/2012
■ Monthly basis
■ T = 389
Models
■ AR and Garch
■ Only one co-variate at a time
■ VAR
■ Combination of covariates: C7,1 , C7,2 , C7,3 , C7,4
■ Distinct functional forms
■ level, log, dlog
■ Total number of models and hence forecasts in each time period
■ N = 318 – this is the number of forecasts in each cross section of the panel
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
The metals price index from 1980 to 2012
Metals price index Real terms
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
Metals price index Real terms
T1 T2
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
Metals price index Real terms
T1 T2
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
Metals price index Real terms
T1 T2
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
Metals price index Real terms
T1 T2
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
T1+1 T2+1
Metals price index Real terms
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
Bias window
T-h-R
Metals price index Real terms
The bias-corrected forecast: learning about the model
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
Bias-corrected 1-step ahead forecast
Metals price index
Real terms
0
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
Metals price index Real terms
Bias-corrected 6-steps ahead forecast
Distribution of the bias
h=1 h=2 h=3
h=4 h=5 h=6
Bias
■ Conley significance test
Bias T-statistic P-value
1-step -0.92 -0.57 0.28
3-steps -2.93 -0.62 0.26
6-steps -6.08 -0.69 0.24
9-steps -9.54 -0.76 0.22
12-steps -12.94 -0.82 0.20
15-steps -16.25 -0.86 0.19
40
120
200
280
2004 2005 2006 2007 2008 2009 2010 2011 2012
BCAF - 1-step ahead
BCAF - 6 steps ahead
Metals price index
The outcome of BCAF
Metals price index Real terms
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Forecasting
Estimation period Forecast period
309 obs 80 obs
Metals price index Real terms
T1
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Forecasting one-step ahead
Estimation period Forecast period
309 obs 80 obs
Metals price index Real terms
T1
Sources: IMF and Vale
Metals price index Real terms
Estimation period Forecast period
310 obs 79 obs
Forecasting one-step ahead
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
310 obs 79 obs
Forecasting one-step ahead
Metals price index Real terms
40
80
120
160
200
240
280
1980 1984 1988 1992 1996 2000 2004 2008 2012
ind
ex
Sources: IMF and Vale
Estimation period Forecast period
388 obs 1 obs
Forecasting one-step ahead
Metals price index Real terms
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
The outcome of forecasting N models 1-step ahead
Metals price index Real terms
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
Comparison of 1-step ahead forecasts
Metals price index
Real terms
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Bias-corrected forecasts Forecasts
50
100
150
200
250
300
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
Metals price index Real terms
The outcome of forecasting N models 6-steps ahead
0
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Sources: IMF and Vale
Forecasts 6-steps ahead
Metals price index
Real terms
0
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
In
dex
Forecasts Bias-corrected forecasts
40
80
120
160
200
240
280
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
ind
ex
Metals price index
AF 1-step ahead
AF 6-steps ahead
Sources: IMF and Vale
Average forecast 1-step ahead and 6-steps ahead
Metals price index Real terms
The weighted average forecast
■ The inverse of the MSE can be used to weight forecasts
■ The higher the error the lower the weight of the forecast
Wi = 1/MSEi
Σ 1/MSEi
40
120
200
280
2004 2005 2006 2007 2008 2009 2010 2011 2012
BCAF
WAF
AF
Metals price index
The outcome of distinct combined forecasts 1-step ahead
Metals price index Real terms
40
120
200
280
2004 2005 2006 2007 2008 2009 2010 2011 2012
BCAF
WAF
AF
Metals price index
The outcome of distinct combined forecasts 6-steps ahead
Metals price index Real terms
Bias
corrected
average
Average
forecast
Weighted
average
(MSE)
Best model
BIC
1-step ahead 11.3 11.0 11.4 11.6*
3-steps ahead 23.7 23.3 24.1 25.3*
6-steps ahead 36.6 37.3 38.6 43.0*
9-steps ahead 44.9 47.4 50.2 57.2*
12-steps ahead 49.7 53.5 59.1 67.7*
15-steps ahead 52.6 58.3 66.6 76.6*
The RMSE of different forecasts
(*) Applying the Diebold-Mariano test , we could not reject the hypothesis that the errors of BIC forecast are different from the errors of the best forecast combination, at 10% level.
Conclusions
■ Combining forecasts proved to be better for predicting the metals price index than any individual model
■ Among the combinations the “Average forecast” performed better in short horizons, while “Bias corrected forecast” outperformed for long horizons
Thank you!