Decision-Making in Oil & Gas - The Good, Bad, and the · PDF fileDecision-Making in Oil...

Decision-Making in Oil & Gas -

The Good, Bad, and the Ugly Reidar B Bratvold

University of Stavanger

Norwegian Inst. of Technology

University of Texas at Austin

EDPN - October 2012 2

Acknowledgement

!  Steve Begg and Matthew Welsh, U of Adelaide

!  Eric Bickel, U of Texas at Austin

!  Bart Willigers, BG Group

!  Current and former graduate students at UoA, UiS, NTNU, and UT Austin

R.B. Bratvold

EDPN - October 2012 3

What is the Basis for my Discussion?

!  Have spent 30 years in oil & gas working as a roughneck, reservoir engineer, geostatistician, executive, and academic

!  A set of formal and informal polls and questionnaires addressing oil & gas professionals (engineers, geoscientists, economists – managers and non-managers)

!  Experience drawn from having provided short courses and consulting services to a large number of oil & gas companies

R.B. Bratvold

EDPN - October 2012 4

Oil & Gas is often referred to as an industry where Decision Analysis is broadly and successfully adopted

Chevron DAS Award

R.B. Bratvold

Raiffa’s Oil Wildcatter

EDPN - October 2012 5

Decision Analysis in Oil & Gas

R.B. Bratvold

!"#$%&'())*+'

!"#$%&',)$'-)'())*+'

EDPN - October 2012 6

Decision Analysis (DA) is a systematic procedure for transforming opaque decision problems into transparent decision problems by a sequence of transparent steps

R.B. Bratvold

Framing, Structuring

Evaluating, Modeling ./&01"$'

Assessing & Deciding ./&01"$'

23)45*635 7'4)890/#:)/');'$))<& '='>3)45&&5&'

Decision Problem

Action

?)/'@)A#3*'

EDPN - October 2012 7

Decision Analysis leverages processes and tools such as the Dialogue Decision Process, Elements of Decision Quality, Strategy Tables, and Sensitivity Analysis !

R.B. Bratvold

!"##$%#&'%'

()*#&+

BB''''''',-%&)'*./&0+

+1'2")#*."'+

3*-4&0++#/*++

5)*6&7"80+

9&*0"'$':+

;-&#&'%0+"2+<&=$0$"'+>4*-$%?+

Framing, Structuring

Evaluating, Modeling Insight

Analysis team

Assessing & Deciding

Insight

Decision-Maker(s)

! however, the main value of decision analysis is not in the specific numbers that are generated in the process but in the ! •  Structured thinking and insight that the process

engenders.

•  The resulting transparency and record.

!"#$%$&'()')*+%$%($%(),&-.(/&0(.&()#/$"1"(#*)2$.+(&3()#4&'($'(

5)6$'7(8"#$%$&'%()'89("1"'(5&2"(3-'8)5"'.)**+9(/&0(.&()#/$"1"(

#*)2$.+(&3(./&-7/.:((( ( ((;&0)28(

R.B. Bratvold 8 EDPN - October 2012

R.B. Bratvold EDPN - October 2012 9

The use of probabilistic modeling in the oil and gas industry has increased significantly over the last 20 years

Has the decision making ability in the oil & gas industry improved as a result of this?

Decision making ability does not appear to be increasing in tandem with uncertainty quantification

To what degree has your organization's ability to quantify uncertainty increased in the last five years? To what degree has your organization’s decision-making ability improved in the last five years?

0%

10%

20%

30%

40%

50%

60%

70%

Regressed/No Change Some Increase Significant Increase No Opinion

Perc

enta

ge o

f Res

pond

ents

Uncertainty QuantificationDecision making

R.B. Bratvold EDPN - October 2012 10

EDPN - October 2012 11

What’s Not So Good?

R.B. Bratvold

!  Lack of decision oriented project management processes

EDPN - October 2012 12 R.B. Bratvold

! Stage-gate project management processes (PMP) are quite common in oil & gas companies

! The purpose of a PMP is to maximize value creation through better decision-making and improved execution

The industry is often using management processes that are not decision oriented

Planning Front End Loading (FEL)

Execution

EDPN - October 2012 13 R.B. Bratvold

In practice, a significant gap exists between the ideals of the PMP and reality

!  Improving decision quality (DQ) is one of the reasons to use a PMP, but too often no one explicitly owns DQ

! During FEL, the emphasis is on improved decision-making -  However, the links between PMP, decision quality, and improved

value creation have been severed at most companies

-  The PMP often gets linked to compliance and assurance

-  It has morphed from a value-maximizing endeavor to a risk-mitigation process. While risk management is important, it is a poor proxy for value maximization

! Most PMP implementations do not include a dialogue decision process -  Advocacy is the most commonly used process despite ample

evidence demonstrating its ineffectiveness

EDPN - October 2012 14

What’s Not So Good?

R.B. Bratvold

!  Lack of decision oriented project management processes

! Poor understanding of the purpose/role of technical work

The main role of a Geoscientist, Engineer, or Economist is to support decision-making

•  Technical work in the oil & gas industry is fundamentally about uncertainty assessment for the purpose of making decisions

•  First priority: Accurate (=unbiased) uncertainty assessment

•  Second priority: Uncertainty reduction – but only if it is value adding in the context of the decision(s) at hand

R.B. Bratvold EDPN - October 2012 15

Many oil & gas professionals consider uncertainty assessments to be of secondary importance – something that can be added after the main

(deterministic) work is done

EDPN - October 2012 16

What’s Not So Good?

R.B. Bratvold

!  Lack of decision oriented project management processes

! Poor understanding of the purpose/role of technical work

! Confusing uncertainty reduction with improved decision quality

We often forget that the goal is to make good decisions which will lead to good outcomes – not to reduce uncertainty

•  Bias in central value (mean): –  e.g. usually optimism (“rose-tinted glasses”),

sometimes pessimism - both destroy value!

•  Bias in width of distribution: –  e.g. assessing the range of uncertainty to be much

less than it really is with respect to your true state of knowledge (overconfidence)

The main enemy of good decision making is uncertainty bias, not the uncertainty being too large

R.B. Bratvold EDPN - October 2012 17

Have we moved from a state of Blissful Ignorance to Uncertainty Induced Confusion?

EDPN - October 2012 18

What’s Not So Good?

R.B. Bratvold

!  Lack of decision oriented project management processes

! Poor understanding of the purpose/role of technical work

! Confusing uncertainty reduction with improved decision quality

! Poor understanding of the nature of probabilities

EDPN - October 2012 19

In the words of someone who knows !

•  "If you don't get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest.“

R.B. Bratvold

Charlie Munger (Vice-Chairman of Berkshire Hathaway Corporation and lifelong business partner with Warren Buffett)

Having a good understanding of probability concepts is essential for good decision making •  Large independent:

–  “The actual performance of our key assets wasn’t even within the P1 to P99 range.”

•  Common response to the question: What is the definition of Expected Value?

–  “The 50/50 of an outcome.”

•  CEO to manager: –  “I want your guarantee that we will not spend more than

the P50 on this project!”

•  Team manager: –  “I prefer to evaluate your project deterministically

because either you’re right or you’re wrong – and if you’re good, you’ll be right!”

R.B. Bratvold EDPN - October 2012 20

The nature of Probability: Probability is a state of mind, not a state of things

Throw a die and hide top face. What is the probability of the top face being a 3?

What is the probability of a 3 now?

Now get information.

R.B. Bratvold 21 EDPN - October 2012

1/6

Has the top face changed? Has the probability that it’s a 3 changed?

Cut away center piece. No Yes 1/3

Uncertainty is a function of what you know. There is no “right” uncertainty (or PDF)!

Uncertainty is in our heads – it’s a function of our state of knowledge

Probability is not an inherent “parameter” of the “system”

•  The systems that we deal with are essentially deterministic, we use probability to express our state of knowledge about them

•  The Die: -  when the die is tossed, there will be one face that comes up

-  in theory, if we knew (precisely) its initial conditions and we could model (precisely) all of the processes involved with tossing it, we could predict how it would land – but that is, practically, impossible

•  And so with geology, oil price etc: -  we use the language of probability to quantify our lack of knowledge

-  for events that have occurred, we could, theoretically, collect information to reduce uncertainty - complete information would resolve it

R.B. Bratvold EDPN - October 2012 22

Probability is subjective (personal) and depends upon your information

Person A Info Person B

Info

Shared Info

Person A PDF Person B PDF

R.B. Bratvold EDPN - October 2012 23

EDPN - October 2012 24

Poor probabilities understanding is also preventing us from learning from our mistakes

R.B. Bratvold

!  Overconfidence survey

!  Oil & gas professionals (mostly engineers, geoscientists, and economists) were asked 10 questions to check how well calibrated they are wrt their lack of knowledge

-  Survey o Total respondents: ~400 o Managers: ~10% o Male: ~75% o Female: ~25%

EDPN - October 2012 25

Questions used to check for overconfidence

R.B. Bratvold

EDPN - October 2012 26

Oil & Gas professionals are as overconfident as anyone else

R.B. Bratvold

CD534)/E*5/$''F0/435#&0/1'$)'$"5'<5GH'

->)$')/'!5<<'4#<093#$5*'

I5#/'7JKL'

EDPN - October 2012 27

People often confuse probabilities with other quantified valued

!  Many participants choose not to answer the questions:

-  “I have no idea.”

-  “How can I provide you with a range when I don’t even know what the correct answer is?”

R.B. Bratvold

!  Probabilities are not point estimates of some quantity

!  Probabilities are used to convey our uncertainty about the quantity

!  If I had precise knowledge about the quantity, I wouldn’t need probabilities at all

Variable

R.B. Bratvold EDPN - October 2012 28

We use models that depend heavily on subjective estimates and yet ignore the research about how well experts estimate and how to adjust for their errors

•  Probability judgments are not simply waiting in our heads to be pulled out when needed

•  Discovering and developing these requires hard and systematic thinking about the important aspects of a decision

•  Human beings are imperfect information processors –  Personal insights about uncertainty and preference can be

both limited and misleading

–  An awareness of human cognitive limitations is helpful in developing the necessary judgmental inputs

A disciplined elicitation and calibration process, not individual genius, is the key to improved assessments

EDPN - October 2012 29

We don’t show much interest in learning from our mistakes •  Prof Daniel Kahneman

–  Winner of 2002 Nobel Prize in Economics

–  Psychologist and Decision Analyst

"  ”The thing that astonishes me when I talk to business people in the context of decision analysis is that you have an organization that’s making lots of decision and they’re not keeping track. They’re not trying to learn from their own mistakes; they’re not investing the smallest amount in trying to actually figure out what they’ve done wrong. And that’s not an accident: They don’t want to know.”

R.B. Bratvold

EDPN - October 2012 30

Even when we are explicitly shown that our thinking is erroneous, we tend to explain it away

R.B. Bratvold

MNO'#/&A53'NKL'PLO'#/&A53'NKJL'LO'#/&A53'PQR' S)3354$'

MNO'#/&A53'NKL'PLO'#/&A53'NKJL'LO'#/&A53'PQR'

EDPN - October 2012 31

Even when we are explicitly shown that our thinking is erroneous, we tend to explain it away

!  People tend to explain away their own faulty thinking after they are shown to be wrong: T  <3(./"(=-"%4&'()'8(,""'(>2&>"2*+(>/2)%"8(<(0&-*8(/)1"(>2&1$8"8(./"(

#&22"#.()'%0"2(

T  ?*./&-7/(./"(>2"8$#."8("1"'.(8$8('&.(&##-29((

o  it eventually will (off on timing)

o  it nearly occurred (the close call)

o  it would have but for ! (the exogenous shock)

!  Bad luck is a far more popular explanation for forecast failure than good luck is for forecasting success

R.B. Bratvold

To improve the quality of your own predictions, keep brutally honest score. Enjoy being wrong, admitting to it and learning

from it, as much as you enjoy being right - Tetlock

EDPN - October 2012 32

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

EDPN - October 2012 33

Decision Analysis Practice Award for 2010 was given to Chevron

R.B. Bratvold

U"5'V540&0)/'W/#<X&0&'23#4:45'WA#3*'0&'10D5/'#//6#<<X'$)'$"5'95&$'*540&0)/'#/#<X&0&'#>><04#:)/Y'#&'Z6*15*'9X'#'>#/5<');'-)405$X'858953&K'U"5'0/$5/$');'$"0&'#A#3*'0&'$)'354)1/0[5Y'>3)8)$5Y'#/*'>69<040[5'1))*'*540&0)/'#/#<X&0&'>3#4:45K''

EDPN - October 2012 34

“Without the commitment to have quality assurance and lookback/feedback as part of the DA process, just doing “DA calculations” probably won’t create much value”

R.B. Bratvold

EDPN - October 2012 35

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

!  Increasingly accepting the existence, if not quantification, of uncertainty

EDPN - October 2012 36

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

!  Increasingly accepting the existence, if not quantification, of uncertainty

!  Islands of advanced uncertainty modeling skills within many oil & gas companies

EDPN - October 2012 37

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

!  Increasingly accepting the existence, if not quantification, of uncertainty

!  Islands of advanced uncertainty modeling skills within many oil & gas companies

! Conferences, Forums, and Workshops focused on DA (sort of)

EDPN - October 2012 38

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

!  Increasingly accepting the existence, if not quantification, of uncertainty

!  Islands of advanced uncertainty modeling skills within many oil & gas companies

! Conferences, Forums, and Workshops focused on DA (sort of)

!  Increasing number of oil & gas focused research papers discussing DA (sort of)

A new journal – SPE Economics & Management – was established 3 years ago

EDPN - October 2012 39 R.B. Bratvold

EDPN - October 2012 40

What’s Good?

R.B. Bratvold

!  2010 DA Practice Award to Chevron

!  Increasingly accepting the existence, if not quantification, of uncertainty

!  Islands of advanced uncertainty modeling skills within many oil & gas companies

! Conferences, Forums, and Workshops focused on DA (sort of)

!  Increasing number of oil & gas focused research papers discussing DA (sort of)

!  Increasing number of consulting firms and software developers specializing in DA for oil & gas

EDPN - October 2012 41

Concluding

R.B. Bratvold

! The oil & gas industry is not bad in terms of its willingness and ability to implement decision analysis but ! also not great

! Other industries are probably as good or better; e.g. -  Automotive

-  Armed forces

-  Pharmaceuticals

-  Others ??

!"#$'#35'X)63'5B>5305/45&+'