INTERFACES Copyright © 1981, The Insiituie of Management SciencesVol. II, No. 4. August 198! 0092-2102/81/1104/0048SOI.25

BUILDING GOOD MODELS IS NOT ENOUGH

Richard Richels

Electric Power Research lnslUute3412 Hillview Avenue. P.O. Box 10412. Palo Alto, California 94303

ABSTRACT. The failureof models to live up to the expeciations and demands of the usercommuniiy has led lo increasing skepticism about ihc usetulne.ss of policy modeling. Thispaper examines reasons for this growing dissatisfaction and considers ways to deal with it.

A group of researchers on policy modeling once observed " . . . there is onething that policy models have in common. Most fall short of their potential asinstruments for the clarification of policy issues and the enlightenment of policymakers'" [Greenberger. Crenson andCrissey, 1976]. There are a number of reasonswhy policy modeling has failed to live up to user expectations. General Glen Kent,formerly head of studies and analysis for the US Air Force, noted several years agothat "decision makers are becoming increasingly annoyed that different analysts getquite different answers to the same problem [Brewer, 1980]." When this happens, itis natural to want to take a closer look at the models and find out why they producesuch puzzling results. Until recently, however, there was very little evaluative in-formation available concerning the quality and appropriate uses of individualmodels. The problem has compounded as the number of models has increased andpolicy makers have become deluged with conflicting claims from model developersanxious to convince users and would-be users of the value of their creation.s.

Users have also found themselves thrust into the uncomfortable position ofmodel defender. Once a model begins to be used as an instrument of debate, it will beseen by some as a threat or challenge and become the subject of investigation[Greenberger and Richels, 1979]. Typical of such challenges is the American PublicGas Association's attempt to discredit the Federal Power Commission's use of amode! in establishing natural gas rates. APGA argued that "the 'economic models'dreamed up by producer-sponsored consultants and untested by cross-examination donot begin to rise to the status of 'substantial evidence' [US Court of Appeals,1977]." The FPC found itself in the position of having to defend its choice of modelsin court. Such situations have fed the growing realization that independent or third-party assessment is critical if policy makers are to have confidence in model results.

But quality control is only part of the problem. Each year countless studies,many containing useful and timely information, gather dust [Fromm. Hamilton, andHamilton, 1974]. Sometimes the fault is with the policy maker for failing to takeadvantage of readily accessible information, but more often the blame is with themodeler. Analyses are presented such that they are more likely to confuse andoverwhelm than inform. This failure to communicate seriously impedes the use ofimportant tools for policy making. The result is that analyses are used more torationalize policy decisions than to serve as a basis fjx rational decision making.

PHILOSOPHY OF MODELING

INTERFACES August 198!

Policy modeling can be enormou.siy valuable in providing general insights into aproblem. But up to now. the educational benefits have accrued almost entirely to themodelers and have failed lo reach the policy arena where they are most needed, Thisis a major weakness with policy modeling. Hogan hit the mark when he wrote that"the purpose of energy modeling is insight, not numbers [Hogan. 1978]." Theproblem is that policy makers are heing presented with numhers, not insights. With-out the rationale to support these numbers, it is small wonder that they have little usefor results that do not conform with their own intuition. If models are to reach theirfull potential, more attention must be placed on communicating the rationale ofpolicy models, not just the results.

INDEPENDENT MODEL ASSESSMENT

The first step to making policy models more useful in policy making has alreadybeen taken; the last few years have seen tremendous growth in independent modelassessment. A number of assessment projects have been established and are provid-ing valuable information essential for the intelligent use of models [Greenberger.1980J. The work done by the Energy Model Analysis Program at the Massachusettsinstitute of Technology is an example of this model-oriented type of analysis [MIT,1978].

In a typical evaluation, the model is examined over the full range of its capabil-ity. The assessors begin by going through the computer code line by line to verifythat the model is as advertised. The validity of the model is then examined in thecontext of the purposes for which the model was constructed or is to be used. Theassessors attempt, through a series of validity checks, to assess the agreement be-tween the model and the real world system being modeled [Gass, 1977].

Verification and validation give the user some indication of the "usefulness" ofa model. The assessors are also concerned with the "usability" of the model. "Amodel is usable if it is understandable and plausible to others than its developers,economic to run on a computer, and accessible to those who wish to use it [Commit-tee on Merchant Marine and Fisheries. 1975]." The usability of a model depends onsuch factors as the quality of its documentation, operating characteristics, and overallportability — all of which are assessed by the model analyzers.

The best time for third-party evaluation is an open question. Independent as-sessment can and does take place at each stage in the modeling process — at thecreation .stage, at the application stage, and even as the model is being used as aninstrument of debate. What is clear is that if careful examination isput off too long,the results are apt to be disappointing. Model users need evaluative informationbefore selecting a model for a policy debate. Without the careful scrutiny of highlytrained analysts, users can only speculate about a model's capabilities, limitations,adequacy, and realism. The likelihood of a poor choice is high. For this reason,independent assessment should first occur early on in the modeling process, beforethe model is brought to the poiicy arena.

Independent assessment should not be considered a "one-shot" activity. Anactive model is in a constant state of evolution. As it is brought to new applications itis modified and refined. Careful examination by third parties also leads tomodifications as the model builder is provided with valuable feedback signals helpfulin correcting and improving the model. If an assessment is to remain timely it mustnot ignore the changing nature of the mode!. Frequent assessment "updates" will benecessary as long as the model remains active.

INTERFACES AugusI 1981 49

But model assessment is only part of the answer. Experience has shown thateven when a model is perceived as "useful" and "usable." it still may not be used.Tbis brings us to tbe second reason wby models bave failed to live up to the hopesand expectations of the user community; the model builder is not communicatingeffectively tbe insights, structure, and understanding available from the model.

User AnalysisThe purpose of policy analysis is to help policy makers make better decisions.

Decision making can be aided by using tbe model as a tool for education andexploration. Tbere is considerable evidence tbat models can be effectively used toimprove understanding. Examples abound as to how a model result whicb initiallyappeared anomalous and implausible led to reassessment of ideas and a deepening ofinsights [Hogan. 1978]. Modelers encounter sucb instances daily. The problem is inmaking tbe insights available to those responsible for policy making. It is not suffi-cient to produce a technical report only understandable to the trained analyst. Aprimary reason why policy analyses go unread is tbey are too long or too bard to beunderstood by the lay reader [Stokey and Zeckhauser. 1978]. Converting technicalresults into a language comprehensible to policy makers is essential if model resultsare to be used.

The difficulty with bringing together people with such diversity in backgroundsas modelers and policy makers is illustrated by the following anecdote:

A congressman who had been persuaded to favor a "soft lechnology" approach lo iheenergy needs of ihe United Stales was discussing projections of demand for electricity wilha modeler. He pointed out ihal with adequate conservation measures, a little modesty in ourlife-siyle, and other steps emphasized by the soft lechnologisls, the rate of growth ofeleclrical demand couid easily be held down to 2 percent per year. "But Mr. Congressman,even at ihe low rate of 2 percent per year, electrical demand will double in thirty-fiveyears," said the modeler. "That's your opinion!" replied the congressman.

[Sachs, 1980]

Although the story may seem somewhat extreme, it is indicative of a veryserious communication gulf, To bridge this gap. it may be necessary to add a user-oriented activity to the modeling process. To connote the intended beneficiary, wewill refer to it as "user analysis." Whereas model assessment focuses primarily onquality-related issues, the emphasis here is on transferring the insights of a policyanalysis to the policy maker. Tbe intention is to insure that the communication of%policy analysis receives as mucb attention as its design and implementation.

User analysis bas several components. Tbe first, the presentation of results inthe context of the policy problem under study, may seem obvious, but it is all toooften overlooked. Tbe modeler should highlight those aspects of his model oranalysis that are relevant to the decision problem of interest to the policy maker. Todo tbis he may wish to employ the techniques of decision analysis, a methodologydesigned specifically for aiding decision making [Raiffa. 1968]. Decision analysisforces the analyst to focus on the decision. The heart of the methodology is thedecision tree which lays out the relevant sequences of decisions and outcomes. Forexample, the decision tree of Figure 1 was used by tbe Synfuels Interagency TaskForce in 1975 to examine alternatives for implementing a synthetic fuel program inthe US [Synfuels Interagency Task Force, 1975]. The squares denote decision nodesand the circles denote chance nodes. The initial decision was a choice among fouraltemative government-financed synthetic fuel programs.

50 INTERFACES Augtist 1981

FIGURE I. SYNTHETIC FUELS DECISION TREE.

The advantage of a decision tree is that it focuses on the sequential nature of thedecision problem. In structuring alternative strategies in a decision tree framework,the analyst works with the decision maker to identify those decisions that must bemade today and separates them from those which can wait and consequently benefitfrom the inflow of additional information. In the synfuel analysis, the decision wasnot whether the US should commit itself to full scale synfuel production, but the sizeof the initial program. Prior to the work of the Synfuel Task Force, the debate hadfocused on the desirability of a full scale synfuel industry. The work of the SynfuelTask Force is credited with focusing the government debate on the real decision athand. The analysis was regarded as a key factor in persuading the Ford Administra-tion to cut back from the President's original goal of a million barrels down to350.000 barrels a day in 1985 [Weyant. 1978].

In presenting tbe results of a policy analysis, the modeler should also indicatethe sensitivity of the decision being studied to the basic assumptions underlying theanalysis. In the synfuel decision analysis, the Task Force assumed that the probabil-ity of a strong oil producers' cartel in 1985 is 50%. but they also showed thesensitivity of the optimal program to tbis assumption. Sensitivity analysis providesthe policy maker with valuable insight into the relative importance of the uncertain-ties complicating his decision. If it turns out that a decision Is very sensitive to aparticular variable, tbe value of better information regarding that variable is bigh,and tbe policy maker may wisb to allocate resources to reduce tbe level of uncer-tainty.

When discussing ihe value of information it is also useful to give some indica-tion as to how accurate information must be to be of value. For example, it is wellknown that assumptions about uranium resources can affect the desirability of ad-vanced nuclear technologies. For tbis reason, the government is spending millions of

INTERFACES August 1981

dollars to reduce uncertainty regarding the uranium resource base. Recent analysishas shown, however, that attaining the large potential benefits from better informa-tion will be difficult, since the information must be extraordinarily accurate to have asignificant impact on the choice of nuclear policy options [Gilbert and Richels,1980]. Such insight can be helpful in determining the poteniial value ofinformation-gathering efforts.

Often model results which appear puzzling at first glance can lead to a reassess-ment of ideas and a deepening of insights. The communication of interesting resultsis the modeler's biggest challenge. To do this it is not only necessary to present theresults, but also the "pathways'" to results. At this stage in the presentation, theoriginal model moves to the background and the focus is on telling an intelligible,intuitive story — one that stands apart from the model. Sometimes the modeler mayfind it useful to rely on a simplified version of the original model to tell the story. TheHogan-Manne fable of the elephant and the rabbit is one such example [EnergyModeling Forum, 1977]. A simple, highly aggregated model was used to illustratethe key concepts that determine the economic impacts of energy policies. The simplemodel which abstracted from the class of large energy economic models was usedquite effectively in communicating the significant energy-economic interactions topolicy makers.

In describing the pathways to results, it is essential to translate from specializedmodeling jargon into a form that is accessible to a wide audience of potential users.Once the essence of the story is conveyed as clearly and concisely as possible, themodeler can then start adding to its complexity by injecting the necessary caveatsabout critical assumptions and model limitations and explaining how they may affectthe conclusions.

This is what Greenberger had in mind when he proposed an "open box" ap-proach to communicating model results to policy makers:

The typical policy model is noi designed for ease of cctmmunicaiion with ihe decisionmaker. Even the model builder may have difficulty comprehending fully the essentialworkings of the model. Ideally, the model should be presenled Io ihe policy maker, not as a"black box" with assumptions and daia feeding into Ihe left and results coming oul fromIhe right, but as an "open box" whose inner workings are suflicienlly Niniplified. expo.sed.and elucidated lo enable ihe policy maker lo irace the chain of causality from input looutput on ai least an elementary fundamental level.

[Greenberger. 1980].

He goes on to point out that "it would be a research project of considerableintellectual content and practical significance to develop open box versions ofselected models of greatest potential interest to policy makers [Greenberger, 1980]."It would also be a research project of enormous challenge. The true art to modelbuilding has always been in keeping the model as simple as possible while stillcapturing the essence of the system under study. The additional constraint of notexceeding the policy maker's comprehension may make the building of the simplemodel more intellectually taxing than the building of the more intricately structuredmodel from which it is derived. Nonetheless, an open box strategy offers the modelera vehicle for making the essence of his argument transparent to the policy maker. Ifsuch an approach could be successfully implemented, the payoff of more effectivecommunication would more than Justify the investment.

52 INTERFACES Augusi 19HI

The final coniponeni to user analysis, identification ot important unansweredquestions, should define the modeler's agenda for new research. User analysis, likemodel assessment, is an interative activity, li is rare that the resolution of one set ofissues does nol lead to a set of new ones. If the debate is ongoing, the modeler willhave a new opportunity to aid decision makers. In fact, if he did a good job the firsttime around, he may even have a mandate for new research.

FINAL COMMENTS

Figure 2 shows the two kinds of model analysis described above, (heir relation-ship Io each other, and to the various stages in the modeling process. Model assess-ment and user analysis are two very different types of activities. Whereas modelassessment is best carried out by individuals independent organizationally of bothmodel builders and model users, user analysis should be done as a normal part of themodeler's work assignment. To allocate the activity to third parties is not onlyinefficient, it breaks Ihe vital link between modeler and policy maker.

FIGURE 2. MODEL ANALYSIS: A NECESSARY BRIDGEBETWEEN THE MODELER AND POLICY MAKER.

(1 [ Suggestions for new work

Model creationand refinement

i

cModel

application

A• • • • ' / T

Model ' /assessment /

• \ .• - . ^ ' • / / .

V / • Verification //y / •Validation/ / . • Documentation/yy • Appropriate uses/ / / " Running effciency' y V • Portability ,,.

//////////////A

i

\

JPolicy

process

y ' / User// analysis/ . ' 1 . ,'

• Decision focus• Value of information• Pathways to results• Assumptions• Um talions• New insights• Unanswered questions

Model Analysis - •- t-

With the support and encouragement of model sponsors and users, these activi-ties can become permanent and important parts of the policy process. For modelassessment, support is needed for the institutionalization of a new professional disci-pline. More laboratories and centers are needed where model analysts can cometogether to practice the art and science of third-party model analysis. Ideally, inde-pendent model assessment will be set up us recurring activity and the continuingwork of permanently funded facilities.

For user analysis, no new institutional framework is called for. Nor do we needto establish a new professional discipline; what is needed is more professional disci-pline. Modelers must pay more attention to converting technical results into languagecompehensible to policy makers. They must put more effort into translating from ihespecialized jargon of economics and Operations Research into a form that is mean-ingful to ihe nonmodeler. This will not happen, however, until the incentives that the

INTERFACES August 1981 53

modeler faces reward effective communication. Understandably, modelers do notreceive the same feelings of elation and accomplishment in writing a report that comewith finally producing plausible results. Sponsors must provide the motivation formore effective communication. From the outset of a study they must be specific as towhat is expected, and should frequently review whether standards are being met. "Ifhelpful guidance is what policy makers desire from the modeling project, then help-ful guidance is what they must supply to the modelers during the conduct of the effort[Greenberger, Crenson, and Crissey, 1976]."

AC KNOWLEDGMENT

I am grateful to Martin Greenberger. Stephen Peck, and Stan Sussman forhelpful comments and suggestions.

REFERENCES

Brewer, G.. 1980. "On Duplicity in the Modeling of Public Problems," Simulation, April.Committee on Merchant Marine and Fisheries, 1975. "Computer Simulation Methods to Aid National

Growth Policy." 94th Congress, GPO. Wa.shington. DC. April.Energy Modeling Forum. 1977. "Energy and the Economy." EMF Report No. t, Stanford University.

Stanford. CA.Fromm. G., Hamilton. W.. and Hamilton. D.. 1974, "Federally Sponsored Mathematical Models:

Survey and Analysis." Report to the National Science Foundation. Washington. DC.Gass, S., 1977. "Evaluation of Complex Models." Computers and Operations Research Vol. 4, pp.

27-35.Gilbert. R. and Richels. R.. 1980, "Reducing Uranium Resource Uncertainty: Is It Worth the Cost?*'

Energy Modeling III: Dealing With Energy Uncertainty. Institute of Gas Technology. Chicago. !L.Greenberger. M.. 1980. "Humanizing Policy Analysis: Confronting Ihe Paradox in Energy Policy Model-

ing." Proceedings of the 1980 Symposium on Validation and Assessment of Energy Models, Gaith-ersburg. MD, May.

Greenberger. M.. 1980. "A Way of Thinking About Model Analysis," Interfaces Vol. 10, No. 2. pp.91-96.

Greenberger. M, Crenson. M.. and Crissey. B.. 1976. Models in the Policy Process. Russell SageFoundation. New York.

Greenberger. M., and Richels. R. 1979. "Assessing Energy Policy Models: Current State and FutureDirections." Annual Revenue of Energy Vol. 4. pp. 467-500.

Hogan, W.. 1978, "Energy Models: Building Understanding for Better Use." Proceedings of the 2ndLawrence Symposium on Systems and Decision Sciences. Berkeley. CA. October.

Hogan, W.. 1978, "The Role of Models in Energy Infonnation Activities." Energy Information. Pro-ceedings of a Workshop held at Stanford University. Stanford. CA. December 1977,

Massachusetts Institute of Technology Energy Laboratory, 1978. Independent Assessmeni of EnergyPolicy Models: Two Case Studies, MIT Energy Lab. Report No. 78-01 I. Cambridge, MA.

Raiffa. H.. \9(iS, Decision Analysis, Addison-Wesley. Reading. MA.Sachs. R.. 1980. "National Energy Policy — A View from the Underside," National Energy Issues -

How Do We Decide, Ballinger Publishing Company, Cambridge. MA.Stokey. E.. and Zeckhauser. R., 1978. A Primer for Policy Analysi.'s. W. W. Norton & Company. New

York.Synfuels Interagency Task Force. 1975. "Recommendations for a Synthetic Fuels Commercialization

Program." GPO. Washington. DC.United States Coun of Appeals, 1977, The Second National Natural Gas Rate Cases, American Public

Gas Association et al.. Petitioners vs Federal Power Commission, Respondent: No. 76^2000 et al.Weyant, J.. 1978. "Energy Modeling in the Policy Process: Oil Price Decontrol and Synthetic Fuels

Commercialization," Proceedings of the 2nd Lawrence Symposium on Systems and Decision Sci-ences, Berkeley, CA, October.

54 INTERFACES Augusi 1981