Problems with Understanding Probability Richard Washington [email protected] (based on Auckland presentation by N Nicholls and the paper:

Problems with Understanding Probability

Richard Washington

[email protected]

(based on Auckland presentation by N Nicholls and the paper:

Nicholls, N. 1999: Cognitive illusions, Heuristic and Climate Predictions, BAMS, 80, 7, 1385-

1397)

Problems with Probability and Group Work

• From the course so far, you can see that probability is an important part of forecasting

• From your experience in seasonal forecasting you will know that group work and discussions are important to the forecaster and user…

• What constrains the effective use of forecasts?

• The literature on how humans make decisions has many lessons for us…..

Impediments to the use of seasonal climate predictions

• Scientific problems (eg., limited skill)• Inappropriate content (eg., categorical forecasts)• External constraints (eg., inability of users to change

decisions)• Complexity of target system (eg., impacts of a predicted

climate anomaly may be unpredictable for some sectors)• Communication problems (eg., confusion due to

multiple forecasts)• User resistance or misuse (eg., user conservatism)• Cognitive illusions and biases e.g. problems with

understanding and communicating probability

Impediments to the use of seasonal climate predictions

• Cognitive illusions and biases e.g. problems with understanding and communicating probability

• Consider an imaginary seasonal forecast which is known to be perfect…..

Would we be able to persuade users to exploit the information?

Would all sectors benefit from the information?

“What forecasts (seem to) mean”, B.Fischhoff, Int. J. Forecasting, 10, 287-303 (1994)

“A forecast is just a set of probabilities attached to a set of future events.

In order to understand a forecast, all one needs to

do is to interpret those two bits of information. Unfortunately, there are problems in

communicating each element, so that the user of a forecast understands what its producer means.”

Cognitive Illusions…

• Cognitive illusions are like optical illusions• …lead to errors we commit without knowing we are doing so• arise from problems in quantifying and dealing with

probability, uncertainty and risk

• Leads to ….– Uncertainties to be denied– Risks to be mismanaged (sometimes overestimated, sometimes

underestimated)– Unwarranted confidence

• Experts and general public both to blame

Some problems with probability

• Framing effect• Availability• Anchoring• Assymetry between losses and gains• Ignoring base rates• Overconfidence• Decision regret• Inconsistent intuition• Belief persistence• Group conformity• Confirmation and hindsight bias

Framing effect

• The way a problem (or forecast) is posed can influence a decision……………..

Framing effect

• East Africa faces an outbreak of an unusual disease which is expected to kill 600 people.

• Two alternative programs to combat the disease have been proposed.

• Which program (A or B) would you favour?– If A is adopted, 200 people will be saved

– If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3 probability that nobody will be saved

Framing effect• East Africa faces an outbreak of an unusual disease

which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour?– If A is adopted, 200 people will be saved– If B is adopted there is a 1/3 probability that 600 people will be

saved and a 2/3 probability that nobody will be saved

• Now, which of these alternatives (C or D) would you favour?:– If C is adopted 400 people will die– If D is adopted there is a 1/3 probability that nobody will die,

and 2/3 probability that 600 people will die

Framing effect(results of surveys in red)

• East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour?– If A is adopted, 200 people will be saved 72% (risk averse)– If B is adopted there is a 1/3 probability that 600 people will be saved

and a 2/3 probability that nobody will be saved 28%

• Which of these alternatives would you favour?:– If C is adopted 400 people will die 22%– If D is adopted there is a 1/3 probability that nobody will die, and 2/3

probability that 600 people will die 78% (risk taking option)

Framing effect(results of surveys in red)

• East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour?

– If A is adopted, 200 people will be saved 72% (risk averse)– If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3

probability that nobody will be saved 28%

• Which of these alternatives would you favour?:– If C is adopted 400 people will die 22%– If D is adopted there is a 1/3 probability that nobody will die, and 2/3 probability that

600 people will die 78% (risk taking option)

• But……. A = CAnd

B = D

Change in ‘framing’ leads to change in decision

Framing effect

• Forecasts given as likelihood of drought may lead to different decisions to forecasts expressed as non-likelihood of wet conditions.

• So, forecasting a 30% chance of drought is likely to cause a different response to forecasting a 70% chance of normal or wet conditions

– Even though the forecasts are the same

Very different responses can be initiated by small changes in wording (framing) of a forecast

Availability

• The ‘availability’ or prominence of events e.g. in the media can bias our perception.

• The high profile of the 1982/83 and 1997/98 El Nino event may bias a user to expect each El Nino event to be like these big events

• e.g. southern Africa 1997/98 event was expected to behave like the 1982/3 event, while other El Nino events were forgotten as a comparison

• Leads to possibility of overestimating unlikely events– e.g. El Nino events in southern Africa lead to drought……

– But El Nino events have been wet in southern Africa and

droughts have happened in without an El Nino………….

SA fears over new El NiñoWeather experts warn of a devastating drought in our region next year, together with floods on the other side of the globe

Southern African weather and agricultural experts this week voiced concern about the early build-up of the El Niño weather phenomenon, which could spark a devastating drought in the region next year.

Two years of soaking rains have given rise to the best crops in years but a severe El Niño, which has the potential to cause droughts on one side of the globe while spurring floods elsewhere, would cut short the profits from this bumper run.

“The Pacific El Niño has developed much earlier than it usually does. It usually gets going in about December and hits full steam in about January, but it is up to full strength right now,” said Mark Jury, a weather specialist at the University of Zululand.

He said the early development could make the El Niño, produced by a disruption in the atmospheric system leading to changes in wind patterns and higher Pacific water temperatures, unstable and cause loss of momentum.

“On the other hand if it keeps going and developing at the rate it is, it could be just as bad as things we had more than a decade ago.”

This century’s worst El Nîño hit in 1982 and 1983, sparking hurricanes, droughts, fires and flooding in more than 15 countries.

Pretoria this week issued a warning that an El Niño was on the way. This follows warnings earlier this month from international scientists that the potential to cause billions of dollars of damage in key agricultural regions worldwide.

Kit le Clus, a director of South Africa’s National Maize Producters’ Organisation, said the early appearance of El Niño created worries over next year’s maize crop, the country’s biggest agricultural commodity.

“The probability of less-than normal maize production next year is very strong,” Le Clus said.

He said El Niño generally brought rains during the spring planting season, which tended to stunt root development. Dry weather in January and February ten hit at a time when plants were in their reproductive and most moisture-reliant phase.

“That’s when you get the devastation,” Le Clus said.

An emerging markets analyst at a local brokerage firm said that in view of a possibly intense El Niño, he was advising caution on investment in Zimbabwe’s stock market.

“I am obviously worried about the effect, particularly in Zimbabwe where the stock market has had such a good year on the back of two consecutive rainy seasons,” the analyst said. The stock exchange soared 90% in US dollar terms in 1996.

“My worry is that if there is a drought the whole thing will come tumbling down again. My view is that I would be treating Zimbabwe with caution at this stage,” he said.

Weather analyst Jury said that in southern Africa the impact of El Niño would be felt hardest in Botswana, due to the development in summer of a high pressure cell over that country.

It would then spread out affecting the northern, central and eastern parts of South Africa and Zimbabwe.

El Niño… or is it El NonesenseThe experts forecast dry weather, and farmers thought they were prepared for it. But then the rains came… A year ago, climate scientists began reporting that they were observing the strongest El Niño ever recorded, with temperatures in the central, and eastern Pacific two to five degrees above normal. El Niño is an abnormal warming of large parts of the central and eastern Pacific which is believed to set in motion major disruptions of weather around the world.Warned of the coming "mother of all El Niños", Southern Africa braced itself for major drought, since that had been the effect of lesser El Niños in the past, Leonard Unganai, of the Drought Monitoring Centre in Zimbabwe, said recently at the ENRICH Southern African Regional Climate Outlook Forum. ENSARCOF, a body of experts and stakeholders, gathered in the Pilanesberg for a post-mortem of their controversial forecasts for the 1997/1998 summer rainfall season.The ENSARCOF team had first met in Kadoma, Zimbabwe, in September 1997 to issue a sober, scientific consensus forecast of the likely impact of El Niño on the coming summer season and counter the scaremongering.

For the first half of the summer (October, November and December), ENSARCOF forecast above normal rainfall for northern Tanzania with about-normal rainfall for October and November in most of the rest of the region, with the possibility of a drier December. They indicated chances of above-normal rain on the western side of the region and below normal on the eastern side.

Cautious

For December to March, the main rainy season, ENSARCOF forecast above-normal rainfall for north-eastern Tanzania, and parts of Zambia and northem Mozambique, drying further south with "significantly below-normal" rain over South Africa, southem Mozambique, Lesotho and Swaziland.These forecasts were certainly of dry conditions, but they were cautious. Yet, all over the region, contingency measures were taken according to the original perception of a catastrophic drought.As Eugene Poolman, research director of the SA Weather Bureau, reported at Pilanesberg, many small farmers did not plant crops at all. In Zimbabwe, ENSARCOF heard, farmers didn't plant at all or. planted late or less, or turned to shorter-growing crops or less productive and less profitable drought-resistant seeds. Livestock was culled.Many banks refused to grant farmers credit. The Zambian national electricity company made plans to reduce power, water officials planned to sink new boreholes, the water development board stopped licensing water rights, and the food reserve agency made plans to import maize.In the event, the lay prophets of doom were proved to be false prophets, and even the scientists were off the mark.The forecast for the first half of the summer was the more accurate, although less so for the south, including South Africa, Lesotho, Swaziland and southern Mozambique,where some eastern areas had above normal rain.

JOHANNESBURG SATURDAY STARAUGUST 1 1998

By Peter Fabricius

International climatologists are reporting the emergence of a La Niña effect on global climate which ought to bring normal or above-normal rain to southern Africa over the coming summer rainfall season. But local scientists warn that the effects of the predicted La Niña will have to be carefully weighed up in conjunction with more local influences, before any firm predictions are made about the weather. La Niña is the more benevolent "sister" of the notorious El Niño - the "Christ Child" named because of its usual emergence at Christmas time. El Niño describes an unusual warming of the waters of the central Pacific Ocean, which is believed to cause major disruptions to global weather, including floods and cyclones and, in southern Africa and elsewhere, droughts. But a severe El Niño last year did not cause the major drought throughout southern Africa over the summer which was predicted, or even the below-normal rain forecast by scientists for large parts of the region during the second half of summer. Local scientists believe this was partly due to ignorance about the mitigating effects of sea temperatures in the Indian Ocean. La Niña is a cooling of the same Pacific waters to below average temperatures, and its usual effect on southern Africa is to increase the chances of normal to above-normal rainfall, according to Nick Graham, director of the experimental forecast division of the International Institute for Climate Prediction in La Jolla, California. He said in an interview that climate predictions were converged on a La Niña effect. "We have a developing La Niña," he said, adding that La Niña normally meant "not dry" conditions for southern Africa. Dr Graham said temperatures in parts of eastern and central Pacific were already down to a maximum three degrees Celsius below normal and the overall temperature was slightly below normal. Climate scientists were expecting it to drop to 1.5 degrees below normal by year's end. "This is a positive sign for southern Africa," he said, while adding that the effects of the abnormal Pacific temperatures on southern Africa were not fully understood. Professor Mark Jury, director of the Climate Impact Prediction Unit at the University of KwaZuLu Natal, cautioned that it was too early to say what effect La Niña would have locally. Since El Niño had not been taken up fully in southern Africa last season, when rains had been closer to normal than predicted, he was unsure that La Niña would produce a wetter summer. Jury said another effect - the Quasi Biannual Oscillation - was kicking in now and could suppress rainfall, causing a drier-than-normal early summer. This could ease, allowing La Niña to act from January, bringing back the rain. The hotter and drier winter this year is a result of the lingering effects of the strongest ever El Niño phenomenon observed last year, according to Jury. Although the El Niño was the strongest ever recorded, with sea temperatures of the central and eastern Pacific two to five degrees above normal, it did not have the widely expected effect of causing major drought in southern Africa as it had in the past. This was partly due to inadequately understood local effects such as the abnormal summer warming of the central Indian Ocean, and cooling of the eastern Indian Ocean, Jury said. These had brought rain during the second half of the summer to the eastern side of the region, mitigating the drying effects of El Niño. When these had gone, El Niño was felt more strongly and the last three to four months had been unusually dry, showing a delayed El Niño impact, Jury said.

Cautions predictions of a wet southern summer as little sister La Niña emerges

Anchoring

• Anchoring refers to the inability to adjust to numbers which are familiar to us and which we start out using – the numerical problem we are considering has this anchor as its reference point…

Anchoring

• What are the last 4 digits of your work phone number?

• Add 3000.

• Remember the number

Anchoring


• Add 3000.

• Do you think the Nile river is longer or shorter (in km) than this number?

Anchoring


• Add 2000.

• Do you think the White Nile river is longer or shorter (in km) than this number?

• Now, how long is the White Nile River?

AnchoringHow long is the River Nile?

• Telephone Nrs + 2000• 2000-2999• 3000-3999• 4000-4999• 5000-5999• 6000-6999• 7000-7999• 8000-8999• 9000-9999• etc

Asymmetry between losses and gains

• First we are offered a bonus of $300. Then choose between:

• Receiving $100 for sure; or

• Toss a coin. If we win the toss we get $200; if we lose we receive nothing.



• Receiving $100 for sure; or PREFERRED






• This time we are first offered a bonus of $500. Then choose between:

• Losing $100 for sure; or

• Toss a coin. If we lose we pay $200; if we win we don’t pay anything.





• This time we are first offered a bonus of $500. Then choose between:

• Losing $100 for sure; or

• Toss a coin. If we lose we pay $200; if we win we don’t pay anything. PREFERRED


• There are big differences in behaviour when faced with loses compared with gains

• We are usually conservative when offered gains• But we are usually adventurous when facing

loses• So the threat of a loss can be dangerous in

seasonal forecasting because we can be too adventurous

Ignoring base rates / prior probabilties

• A taxi was involved in a hit and run accident at night. Two taxi companies, the Green and the Blue, operate in the city.

• 85% of the taxis are Green and 15% are Blue• A witness identified the taxi as Blue• The witness identifies the correct colour 80% of

the time and fails 20% of the time• What is the probability that the taxi was Blue?

Ignoring base rates

• Probability that witness saw Green taxi and called it Blue is 0.85 X 0.20 = 17% of all cases

• Probability that witness saw a Blue taxi and correctly identified it is 0.15 X 0.80 = 12% of all cases

• More likely that the witness saw a green taxi and thought it was blue!

• Bayes theorem

Ignoring base rates

• When subjects respond to this question, the ignore the base rates (in this case: how many blue or green taxis operate in the city)

• Instead, they rely on the reliability of the witness

Ignoring base rates

• Imagine that a model (with an accuracy of 90%) predicts that my farm will be suffering from drought in October-December 2002.

• Historically there is a 10% chance of drought.

• The model has no bias (i.e. it will forecast droughts with 10% frequency)

• What is the chance that October-December 2002 will be dry (given the model forecast is for a dry year)?

Ignoring base rates

• But the chances that October to December 2002 will be dry (given the model forecast) is only 50%

• P(Non-Dry Year) = 90%• P (Dry Year) = 10%• P (Correct forecast) = 90%

• P (Dry Year/Warning) = P(dry) x P(correct)

P (dry) x P (correct) + P (not dry) x P (incorrect)•

= 0.1 x 0.9 / 0.1 x 0.9 + 0.9 x 0.1• =50%• Despite the accuracy of the model, the chances of drought are still 50/50 (a

coin toss!)

Overconfidence• "We don't like their sound. Groups of guitars are on the

way out.” --Decca Recording Company executive, turning down the Beatles, 1962

• “Mugabe will be gone by December 2001” – Morgan Tsvangirai June 2001

• "With over fifty foreign cars already on sale here the Japanese auto industry isn't likely to carve out a big share of the market for itself."--Business Week, 1968

• The 1997/98 El Nino event will be dry in southern Africa!

• “A La Nina event cannot develop this year – the Pacific is too warm” – Mason and Graham, April 1998

Overconfidence

• Which of these causes of death is most frequent (and how confident are you)?– All accidents, or heart attacks?– murder, or suicide?

Overconfidence

• Select from each pair the most frequent cause of death (and decide how confident you are):– All accidents, or heart attacks?– Homicide, or suicide?

• First alternative is selected with great confidence by most respondents.

• The correct answer is the second in each pair.

Overconfidence

• Overconfidence is a common problem in forecasting

• It is important to – consider all the reasons that the forecast

could be wrong– Consider alternative forecasts– Appoint someone to argue the opposite case

Overconfidence

– Remember:

– Nature does not make it a priority for us to understand her………..

– Climate is a complicated system

Inconsistent insight (human judgement)

• Many decisions are reached by judgement, based on intuition e.g. farmers facing a drought will rely on their own experience in their response

• But human judgement based on intuition is often poor

• It is often better to substitute an objective decision making scheme than to use human judgement

Belief persistence

• Primacy effect : Judging things by the information that we receive first…

Belief persistence

• Primacy effect : Judging things by what comes first…

• Asch (1946) gave subjects a list of adjectives describing a person, such as – “intelligent, industrious, impulsive, critical,

stubborn, envious”, or – “envious, stubborn, critical, impulsive, industrious,

intelligent”.

• The impressions of the person were more favourable given the first list than the second!

Belief persistence

• These are problems that bias a forecaster or a user of a forecast…..

• If the first forecast we issue is correct, we are likely to become biased about the forecast skill

• Forecast producers, faced with forecasts from several models, may give more weight to the first forecast he/she receives

• Inertia may lead forecasters to ignore evidence which contradicts their belief in an existing forecast

Confirmation bias

• Can you work out the rule I have in my head for producing 3 numbers?

• “2 4 6” are 3 numbers that obeys the rule.• Suggest other number sequences - I will tell you

if they conform to the rule or not.• Stop when you think you know the rule.

Confirmation bias

• Wason (1960): “2 4 6” test.• Rule is: Three increasing integers.

• “The human understanding when it has once adopted an opinion draws all things else to support and agree with it” (Francis Bacon, 1620)

Group conformity- the Asch experiment

• Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered “B”.

Test lineA B C



• If person in front of subject said “A”, the error rate increased from 1% to 2%




• If two people ahead of subject said “A”, error rate increased to 13%.





• If three people ahead of subject said “A”, error rate increased to 33%.





• If three people ahead of subject said “A”, error rate increased to 33%.

• If, as well, subject was told a monetary reward for the group as a whole depended on how many members of the group gave the correct answer, the error rate increased to 47%.

Group conformity

• Forecasters and users (e.g. farmers discuss information in groups)

• Decisions taken in groups may easily be wrong

• Group dynamics take over and the group may often reach concensus too quickly

• Groups tend to support the group leader, regardless of their opinion.

• We can all think of such examples!

Hindsight• We tend to remember past events when we have been

correct in our forecasts• We tend to forget when we have been wrong• This leads to increased confidence which is false• e.g. 1997/8 ENSO: Experimental Long Lead Bulletin

contained few warnings of El Nino: – But….– Kerr, R.A. 1998: Models win big in forecasting El

Nino, Science, 280, 522-523.– Most model forecasts were very poor…

Reducing the effect of cognitive biases on the use of climate predictions

• De-bias groups preparing forecasts, to reduce over-confidence• Avoid “anchoring” media reports and thus users (eg., to the

1982/83 El Niño)• Determine how users interpret probabilities (worded and

numerical) - then use these interpretations in forecast preparation• Write forecasts to avoid possible “framing” biases (eg., include

multiple versions of forecasts, framed in different ways)• Do not combine forecasts subjectively or intuitively• Avoid base rate underestimate bias (eg., include specific base rate

information in forecast)

References

• Nicholls, N., 1999. “Cognitive illusions, heuristics, and climate prediction”. Bulletin of the American Meteorological Society, 80, 1385-1397.

• Piattelli-Palmarini, M., 1994. Inevitable illusions. Wiley, 242 pp.

• Plous, S., 1993. The psychology of judgement and decision making. McGraw-Hill, 302 pp.

Documents

Problems with Understanding Probability Richard Washington [email protected] (based on Auckland presentation by N Nicholls and the paper: