Upload
aubrey-austin
View
218
Download
0
Embed Size (px)
Citation preview
Asking the Oracle: Introducing Forecasting Principles into
Agent-Based Modelling
GRASIA, Universidad Complutense de Madrid
INSISOC, Universidad de Burgos
GUESS, Universidade de Lisboa
Samer Hassan
Javier Arroyo
José M. Galán
Luis Antunes
Juan Pavón
Samer Hassan WCSS 2010 2
Contents
A recurrent issue
The Field of Forecasting
Forecasting Principles into ABM
Conclusions
Photo
Samer Hassan WCSS 2010 4
A recurrent issue… in ABM
How many times have we heard…
Are accurate predictions possible within ABM?
Should predictions be the main aim of ABM?
Is ABM mature enough to make proper predictions?
Stakeholders want predictions: is ABM an answer?
Samer Hassan WCSS 2010 5
A recurrent issue… in SIMSOC
“Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact
forecasts?”
Samer Hassan WCSS 2010 6
A recurrent issue… in SIMSOC
“Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact
forecasts?”
“The response, once misunderstandings were sorted out, was several accounts of reasons why policy impacts could not be forecast. The criteria I suggested for deeming a forecast to be correct was the correct forecast of the timing and direction of change of specified indicators.” Scott Moss, SIMSOC list, June 2009, “what is the point?”
Samer Hassan WCSS 2010 7
A recurrent issue… in JASSS
Joshua Epstein (2008): Prediction is one possible aim for ABM… among 16 others
• Explanation• Guiding data collection• Raise new questions• Challenge theories• …
“Explanation does not imply Prediction”• Tectonics explain earthquakes but cannot predict them
Samer Hassan WCSS 2010 8
A recurrent issue… in JASSS
Joshua Epstein (2008): Prediction is one possible aim for ABM… among 15 others
• Explanation• Guiding data collection• Raise new questions• Challenge theories• …
“Explanation does not imply Prediction”• Tectonics explain earthquakes but cannot predict them
Thompson & Derr (2009): “Good explanations predict” An explanatory model is valid only if it predicts real behaviour
Samer Hassan WCSS 2010 9
A recurrent issue… in JASSS
Klaus Troitzsch (2009) Epstein & Thompson discuss different “Prediction levels”:
1) Prediction of the kind of behaviour of a system, under arbitrary parameter combinations and initial conditions
o Earthquakes occur because X and Y
2) Prediction of the kind of behaviour of a system in the near future
o Region R is likely to suffer earthquakes in the following years because X and Y
3) Prediction of the state a system will reach in the near future1) Region R will suffer an earthquake of power P in expected day D with
confidence C
1) “Explanation does not imply 3rd level Prediction” “Good explanations usually imply 1st or 2nd level”
Samer Hassan WCSS 2010 10
A recurrent issue
Agent-Based Modelling has multiple aims…
…but still modellers might seek prediction…
How could we help them?
Samer Hassan WCSS 2010 11
Contents
A recurrent issue
The Field of Forecasting
Forecasting Principles into ABM
Conclusions
Samer Hassan WCSS 2010 12
The field of Forecasting
Forecasting A field focused on the study of prediction
• Specially aiming 3rd level
30 years experience
• Consolidated (journals, conferences)
• Formalised
• “Forecasting experiment”
Samer Hassan WCSS 2010 13
The field of Forecasting
Using ABM as a Forecasting tool
Setting up a forecasting experiment: Split data in two sets
“Objective” error measures
Compare the model
Fair Comparison
Samer Hassan WCSS 2010 14
The field of Forecasting
Using ABM as a Forecasting tool
Setting up a forecasting experiment: Split data in two sets
• Training set (in-sample): calibration
• Test set (out-of-sample): validation
“Objective” error measures
Compare the model
Fair Comparison
Samer Hassan WCSS 2010 15
The field of Forecasting
Using ABM as a Forecasting tool
Setting up a forecasting experiment: Split data in two sets
“Objective” error measures
• Error(t)= forecasted(t) - actual_value(t)
• Aggregated Error of time series:
• Root Mean Square Error
• Mean Absolute Error…
Compare the model
Fair Comparison
Samer Hassan WCSS 2010 16
The field of Forecasting
Using ABM as a Forecasting tool
Setting up a forecasting experiment: Split data in two sets
“Objective” error measures
Compare the model
• Benchmarks: other models, not necessarily ABM
• Naïve method (at least): V’(t+1)= V(t)
Fair Comparison
Samer Hassan WCSS 2010 17
The field of Forecasting
Using ABM as a Forecasting tool
Setting up a forecasting experiment: Split data in two sets
“Objective” error measures
Compare the model
Fair Comparison
• Representative, large sample of forecasts
• Ex-ante: forecast of (t+1) uses info available until (t)
• Out-of-sample: not include training data in comparison
Samer Hassan WCSS 2010 18
Contents
A recurrent issue
The Field of Forecasting
Forecasting Principles into ABM
Conclusions
Samer Hassan WCSS 2010 19
Forecasting Principles into ABM
Principles of Forecasting Armstrong (2001) with 40 authors Summarising the best practices
Selection of subset for ABM
Six topics: Modelling Process Use of data Space of solutions Stake-holders Validation Replication
Samer Hassan WCSS 2010 20
Forecasting Principles into ABM
Modelling Process Decompose the problem into parts
• Bottom-up approach + combination of results
Structure problems that involve causal chains• Results of a (sub)model as input for next one• More accurate than global simulation
Consider the use of adaptive forecasting models• ABM as adaptive systems
Samer Hassan WCSS 2010 21
Forecasting Principles into ABM
Data-driven modelling Use theory to guide the search for information on
explanatory variables
• Reduce complexity pruning design space in advance
Use diverse data sources
• Increase of data reliability
Keep forecasting method simple
• KISS
Select simple methods unless empirical evidence calls for a more complex approach
• KISS + gradual increase of complexity on demand
Samer Hassan WCSS 2010 22
Forecasting Principles into ABM
Space of solutions Identify possible outcomes prior to making forecasts
• Avoid biases
Design test situations to match the forecasting problem
• Put forward scenarios to rehearse policies
Adjust for events expected in the future• Expectability should guide design space exploration and
what-if questioning
Samer Hassan WCSS 2010 23
Forecasting Principles into ABM
Stake-holders and Policy-makers Obtain decision makers' agreement on methods
• Ideally participatory simulation
Ask unbiased experts to rate potential methods• Emphasising their role
Test the client's understanding of the methods• Including limitations of the model
Establish a formal review process to ensure that forecasts are used properly
• Policy deployment should be controlled
Samer Hassan WCSS 2010 24
Forecasting Principles into ABM
Validation List all the important selection criteria before
evaluating methods• Temptation of redefining criteria to fit the outcomes
Use “objective” tests of assumptions• Quantitative approach to test assumptions when possible
Use extensions of evaluations to better generalise about what methods are best for what situations
• Generalisation leads to applicability; based on what-if scenarios
Use error measures that adjust for scale in the data• Error measuring is as important as accuracy of data
Establish a formal review process for forecasting methods
• Ensure verification, replication, trust
Samer Hassan WCSS 2010 25
Forecasting Principles into ABM
Replication Compare track records of various forecasting methods
• The role of replication for ABM verification
Assess acceptability and understandability of methods to users
• Sharing of models & code
Describe potential biases of forecasters• From both modellers and stakeholders• How sensitive is the model to those biases?
Samer Hassan WCSS 2010 26
Contents
A recurrent issue
The Field of Forecasting
Forecasting Principles into ABM
Conclusions
Samer Hassan WCSS 2010 27
Conclusions
The choice of Agent-Based Modelling implies Interest in the “what” is going to happen (Prediction) Interest in “how” the phenomenon occurs (Understanding)
Prediction (3rd level) is a hard job Financial crisis Climate change
Forecasting Principles Best practices, not a solution Helpful in seeking the “what”
Samer Hassan WCSS 2010 28
Thanks for your attention!
Samer Hassan
Universidad Complutense de Madrid
Samer Hassan WCSS 2010 29
Contents License
This presentation is licensed under a
Creative Commons Attribution 3.0 http://creativecommons.org/licenses/by/3.0/
You are free to copy, modify and distribute it as long as the original work and author are cited
Para ver esta película, debedisponer de QuickTime™ y de
un descompresor TIFF (sin comprimir).