SO WHO IS THE BEST PREDICTOR IN THE
AUDIENCE?
PRIZE: A REAL CRYSTAL BALL
BOTTLE OF CHAMPAGNE
PART 1: MAKES SOME PREDICTIONS
PART 2:LIVE EXPERIMENT
WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE
SELLING PRICE OF THE NEW* IPAD MINI 2 WEEKS PRIOR TO ITS
LAUNCH HOW CLOSE DID THEY GET?
WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%
*Note we told them the existing selling price of the old model
WHAT PROPORTION OF WINE DRINKERS IN THE UK
PREFER RED WINE?BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE
WINE
PREDICT HOW MANY RESEARCHERS CHECK
THEIR EMAIL BEFORE BREAKFAST?BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS
England v Montenegro+3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland+3 +2 +1 0 -1 -2 -3
PREDICT WHAT MARGIN OF VICTORY OUR UK
PANELISTS PREDICTED FOR THESE 2 FOOTBALL
MATCHES
The CXO Advisory group
gathered 6,582 buy or sell
predictions from 68 different
investing gurus made between
1998 and 2012, and tracked the
results of those predictions. How
accurate were they?
WHAT % WERE CORRECT?
BACKGROUND
Gamification More prediction protocols in surveys
Fostered an interest in the science of prediction Led to a
series of dedicated prediction experiments Exploration of
the world of prediction market trading Prediki
30+ Primary research experiments
500+ Predictions analysed60+ Prediction markets v traditional research comparisons
THE TYPES OF EXPERIMENTS WE HAVE RUN
• Betting on the future of brands
• Predicting why people buy things
• Predicting the behaviour of other people
• Predicting the price of things
• Predicting the election prospects of political parties
• Predicting football match results
• Predicting the outcomes of TV game shows
• Predicting the success of adverts
• Predicting future sales of products
• Predicting the future more generally
20%
42% 43%
32% 30%
38%
11%
21% 21% 22% 22%
29%
0%5%
10%15%20%25%30%35%40%45%50%
Consumerpurchasingestimates
Observedbehaviour of
others
Observedopinion
Forecast Priceprediction
Guesswork
Correct prediction Random chance
WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?
SOME OF US ARE BETTER AT PREDICTING
0%
20%
40%
60%
80%
100%
120%
140%
160%
180%
score 1 score 2 score 3 score 4 score 5 score 6 score 7
Index of Prediction performance over 7 waves of experiments
top 100
bottom 100
𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 𝑄𝑢𝑎𝑙𝑖𝑡𝑦=
𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 × 𝐸𝑓𝑓𝑜𝑟𝑡 × 𝑂𝑏𝑗𝑒𝑐𝑡𝑖𝑣𝑖𝑡𝑦 × 1 − 𝐷𝑖𝑓𝑓𝑖𝑐𝑢𝑙𝑡𝑦 × 𝑅𝑎𝑛𝑑𝑜𝑚𝑛𝑒𝑠𝑠
Note: Not directly dependent on sample size
Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election
16 IS A CROWDJed Christianson, University of Birmingham calculates
LESS ABOUT SAMPLE SIZE MORE ABOUT
SAMPLE DIVERSITY & INTELLIGENCE
1906 Plymouth County fair
Actual weight = 1198 lb
Median average guess = 1207lb
Error = <1%
WHAT WE KNOW
THE WISDOM OF CROWDS
CROWD WISDOM IS BASED ON FILTERING
THE SIGNAL FROM THE NOISEEach person’s prediction is made up of 2 components: information & error.
If each individual’s judgement is independent & unbiased then the error
will largely cancel itself out and the aggregation process then distils off the
inherent knowledge.
Actual selling price = £319
Median average guess = £316
Error = 1% SCORE: 1 POINT
2013 GMI online sample
WHAT WE KNOW
THE WISDOM OF CROWDS
2014 GMI online sample
Actual weight = 550kg
AN UNWISE CROWD
Median average guess = 350kg
Error = 36%
CROWD WISDOM CAN BE A BIT BEHIND THE TIMES
-44%
-34% -33%-30% -28% -26% -25% -25%
-17%-14%
2%5%
9% 9%
21% 23%
-50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
Examples of price predicition errors
Average 9% price lag
94%87%
0%10%20%30%40%50%60%70%80%90%
100%
Women Men
Price prediciton accuracy
Men less price savvy
“If each individual’s judgement is independent &
unbiased then the error will largely cancel itself out”
THE PROCESS OF MAKING PREDICTIONS
IS LITTERED WITH COGNITIVE BIASES
HOW THIS EFFECT CORRUPTS PREDICTIONS….
54%
46%46%
54%
0%
10%
20%
30%
40%
50%
60%
White Red
What percentage of people do you think prefer white wine?
What percentage of people do you think prefer red wine?
V
20% SHIFT IN PREDICTION
Predict the chances of it raining 5 days in advance
IF RAINING TODAY +20°%
Score If you predicted correctly = 1 point
STUDYING THE IMPACT OF NUDGE EFFECTS: THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER
2%
6%
11%
15%
20%
Personalpreferences
Self evidentpredictions (e.g.ad evaluation)
Factual (requiringknowledge)
Inverted personalpreference (e.g.pedicting relativelevels of dislike)
Complexestimates
Nudge influence by prediction task
Source: GMI research 2014
THE LESS CERTAIN PEOPLE ARE AND THE HARDER THE PREDICTION,
THE MORE WE RELY ON OTHER PEOPLE’S OPINIONS
- 10 20 30 40 50 60 70 80 90
100
Believers
Believe this How many people believe this
- 10 20 30 40 50 60 70 80 90
100
Non believers
Don’t believe How many people don't believe this
those holding minority opinions assume more people agree with them than those holding the majority opinion: is this the definition of
delusion?`
WHAT WILL THE WORLD BE LIKE IN 2050?
55% 68%
YOUR PREDICTIONS
SCORE: +/-5% 2 POINT
+/-10% 1 POINT
50%
20%
I check my emails
before breakfast
I don't check my
emails before
breakfast
Prediction of how many other
people check emails before
breakfast*
*Source: office poll! 80% OF MARKET RESEARCHERS
England v Montenegro
+3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland
+3 +2 +1 0 -1 -2 -3
PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!
SCORE: CORRECT = 1 POINT PER QUESTION
0%
10%
20%
30%
40%
50%
60%
new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2
Newcastle Liverpool
0%
10%
20%
30%
40%
50%
chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3
Chelsea Cardiff
0%
10%
20%
30%
40%
50%
man by 3 man by 2 man by 1 draw south by1
south by2
south by3
Southhampton Man U
0%
10%
20%
30%
40%
50%
man by 3man by 2man by 1 draw south by1
south by2
south by3
Southhampton Man U
FOOTBALL SCORE PREDICTIONS
0%
10%
20%
30%
40%
50%
60%
70%
80%
Conservative
Government
Labour Pary
Government
Liberal Democrats
Government
Conservative &
Liberal Coalition
Labour and Liberal
Coalition
Conservative,
Liberal, & UKIP
Coalition
Conservatives Labour LiberalDemocrats UKIP
PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION
SOCIAL COGNITIVE BIASES RENDER
PREDICTIONS ABOUT OUR OWN
BEHAVIOUR PARTICULARLY DIFFICULT
Will you tidy up after the meeting?
Yes = 50%
TIDIED UP =13%
Predict how many will tidy up?
= 15%
WE ARE OFTEN TOO TIED UP IN THE
DETAIL TO SEE THE BIGGER PICTURE
WILL THE MARRIAGE LAST?
Yes/No
Parents much better than the married
couples at predicting this
Source: Queens University Canada
UNABLE TO SEPARATE THE SIGNAL
FROM THE NOISE
Will the Market Research industry be bigger
or smaller in 10 years time?Yes/No?
0.9
0.92
0.94
0.96
0.98
1
Experts Dillettantes
(non experts)
Chimps
(random
guesses)
Future Predictions accuracy
CA
LIB
RA
TIO
N
Highly recommended reading
PHILIP TETLOCK STUDYING 15,000
GEO-POLITICAL PREDICTIONS
0
0.01
0.02
0.03
0.04
0.05
0.06
Experts Dillettantes
(non
experts)
Chimps
(random
guesses)
Descrimination
Philip Tetlock
EXPERT POLITICAL JUDGEMENT
HOW MANY INVESTMENT GURUS STOCK
MARKET PREDICTIONS WERE CORRECT?
SCORE BELOW 50% = 1 POINT
48%2% less accurate than a coin toss!
-200%
-150%
-100%
-50%
0%
50%
100%
150%
200%
250%
300%
350%
Evaluation of 20 different ads
Monadic rating
0.89 correlation 5x differentiation
MAKE IT REWARDING
WEIGHT BASED ON CONFIDENCE
33%37% 36%
44%
0%
10%
20%
30%
40%
50%
Total Guess Have a Hunch Fairly Sure Very Confident
Prediction accuracy
MONEY BET IS A PROXY
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-3 -2 -1 +1 +2 +3
Bet amount: correlation with outcome
USE PREDICTION MARKETS
IOWA ELECTRONIC MARKET 480/590 OUT PREDICTED THE BEST POLL
SAMPLES OF UNDER 20
32 HEAD TO HEAD EXPERIMENTS
A survey based approach with random cells* of 15
participants who were asked to predict which products would
sell more
vs.
15 people prediction markets trading – asked to buy or sell
variable amounts to create a confidence weighted market
* Using Montecarlo simulation technique we aggregated the predictions of 10,000 randomly
selected group of 15 participants from a larger sample to make this
37%
55%
65%
0%
10%
20%
30%
40%
50%
60%
70%
Random guess Micro survey(sample of 15)
15 people predictionmarket trading
HEAD TO HEAD COMPARISON
Source: GMI/Prediki based on 32 direct head to head comparisons
OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY
BE SET IN STONE IF NO NEW INFORMATION ISADDED
ADDING MORE PEOPLE
AFTER A CERTAIN POINT
DOES NOT CHANGE THE
RESULT
37%
55%
65%69%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Random guess Micro surveysample of 15
15 peopleprediction
markets trading
Standard survey:sample of 200
HEAD TO HEAD COMPARISON
DIALECTICAL BOOT STRAPPINGENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS
NEEDED TO SOLVE PREDICTION CONUNDRUMS
Example = Board room decisions
Useful reference: Herzog and Hertwig (2009)
37%
55%
65%69%
81%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Random guess Micro survey(sample of 15)
15 people Microprediction market
trading
Standard survey(sample of 200)
Micro predictionmarket with sharedinformation & free
comments
THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET
TRADING SYSTEM
EFFECTIVE USE OF PREDICTION MARKETS
• Incentivise - ideally with real money!
• Allow active & dynamic trading
• 16 is a crowd
• Share as much information as possible
• A moderator is important to stimulate debate and share
information
• Divide the herd: run multiple micro markets