Delphi Dr. Camille DeYong Slide 1

Oklahoma State University

Expert Opinion

• Three circumstances– No historical data– Impact of external factors is more important

than the factors that governed the previous development of technology

– Ethical or moral considerations dominate

Delphi Dr. Camille DeYong Slide 2

Oklahoma State University

How to obtain expert opinion?

• Committees

• Advantages– Synergy– Total is greater than the sum of the parts

Delphi Dr. Camille DeYong Slide 3

Oklahoma State University

Disadvantages

• Misinformation

• Social pressure

• Group takes on a life of its own

• Repetition of arguments can influence opinion

• Vulnerability to dominant individuals

• Vested interests

• Entire group may share common biases

Delphi Dr. Camille DeYong Slide 4

Oklahoma State University

How to counteract this?

• Structured group processes– Delphi Process– Nominal Group Technique

Delphi Dr. Camille DeYong Slide 5

Oklahoma State University

Delphi Process

• Developed at the Rand Corporation as a means of extracting opinion from a group of experts.

• First dealt with a series of technological forecasts– Not just a TF method, however

Delphi Dr. Camille DeYong Slide 6

Oklahoma State University

Delphi Process

• Conducted in rounds

• Panel (participants) are not physically together

• Panel answers directed survey questions

• Moderator has responsibility for compiling and feeding back the input.

Delphi Dr. Camille DeYong Slide 7

Oklahoma State University

Three characteristics

• Anonymity

• Iteration with controlled feedback

• Statistical group response

Delphi Dr. Camille DeYong Slide 8

Oklahoma State University

Anonymity

• Avoids the possibility of associating responses with specific individuals

• Participants can change their minds without publicly admitting he/she has done so.

Delphi Dr. Camille DeYong Slide 9

Oklahoma State University

Iteration

• Interaction is conducted through responses to questionnaires

• Moderator extracts pertinent information

Delphi Dr. Camille DeYong Slide 10

Oklahoma State University

Statistical Group Response

• What kind of output do most groups produce?

• Delphi– Presents a statistical response– Opinions of entire group– Statistics that describe both the “center” and the

“spread”• Typically “quartiles”

Delphi Dr. Camille DeYong Slide 11

Oklahoma State University

Round 1

• Typically is completely unstructured– Anticipate events or trends in the technology of

interest– Advantages/disadvantages to this approach

• Questionnaires returned to moderator to compile– Similar items are combined, some dropped,

events stated as clearly as possible– This is the questionnaire for the second round

Delphi Dr. Camille DeYong Slide 12

Oklahoma State University

Round 2

• Consolidated list of events

• Asked to estimate time of occurrence

• Moderator prepares statistical summary– Median date– Upper and lower quartile dates

Delphi Dr. Camille DeYong Slide 13

Oklahoma State University

Round 3

• Third questionnaire consists of the set of events and the statistical summary

– Events, medians and quartiles

• Asked to prepare new forecasts for each event

• If their forecasts fall in the upper/lower quartiles, they are asked to support their position.

Delphi Dr. Camille DeYong Slide 14

Oklahoma State University

Round 3 (cont’d)

• Moderator prepares a statistical summary of forecast and summary of arguments

• Arguments/positions/objections are advanced just as they would be in a face-to-face meeting but,– interaction is anonymous and written.

Delphi Dr. Camille DeYong Slide 15

Oklahoma State University

Round 4

• Questionnaire consists of – Lists of events– Medians/quartiles– Summary of arguments for changing the

forecasts

Delphi Dr. Camille DeYong Slide 16

Oklahoma State University

Round 4

• Panelists are asked to consider the reasons offered for changing the estimates and make new forecasts for each event.

• May be asked to justify position

• Moderator again computes medians and quartiles (and summarizes comments if necessary).

• Looking for “stability”

Delphi Dr. Camille DeYong Slide 17

Oklahoma State University

Outcome

• Events/trends

• Dates - usually use the median date in the fourth round

• Understanding of factors panelists believe are important and may affect the forecast.

Delphi Dr. Camille DeYong Slide 18

Oklahoma State University

Delphi vs. Committees

• Structured process for discussion and disagreement

• Panel members may or may not shift– Can do so without “losing face”

• Delphi generally results in a convergence of the panel estimates.– this is less frequently an outcome of an

unstructured committee

Delphi Dr. Camille DeYong Slide 19

Oklahoma State University

Variations on Delphi

• Providing an initial list of events

• Beginning with a context

• Number of rounds

• Multiple dates

• Computerized

• Partial Anonymity

Delphi Dr. Camille DeYong Slide 20

Oklahoma State University

Important issues

• Group process• Efficient• Precision• Selection of panel members• Survey/questionnaire design

Delphi Dr. Camille DeYong Slide 21

Oklahoma State University

Success factors

• Selection of experts– Pay them?– Contact them personally– Explain the method

• Easy survey

• High response rate