Policy Sciellces 15 (1982) 167 181 167 Elsevier Scientific Publishing Company, Amsterdam - Published in The Netherlands

Interpreting M o d e l Results -

Examples From an Energy M o d e l BRITA SCHWARZ and JOHN HOAG

EFI, T/w Economic Research Institute at the Stockholm School o f Economics, Box 6501, S-113 83

Stockholm, Sweden

ABSTRACT

With the increasing use of complex computer models for high-level policy decisionmaking, the problem of correctly interpreting and communicat ing model results becomes a more general concern. This paper traces misconceptions about the use of models to the existence of different conceptions of the term "model." Policy models are quite often less theory-based than models in the traditional disciplines, especially in cases where the policy models deal with the long-term developments of socioteehnical systems. The authors examine the use of an example of one such model. Generalising from the authors ' experiences in other fields of application, e.g., global modeling, the problems of interpreting model results are discussed. The proper use of future-oriented policy models is clarified by the introduction of typologies implying distinctions, e.g., between forecasting, "what-if," and learning models, and between different "levels" of results, viz. model outcomes, model inferences and policy-issue oriented interpretations.

I Introduction

There has been an increasing use of large-scale mathematical models in policy analy- sis, resource allocation and strategic planning. This development was most conspicu- ous during the sixties in defence (Sanders, 1973) and in urban planning (Lee, 1973), and has since spread to many other fields such as global modeling, short-term econometric forecasting and energy modeling (Greenberger et al., 1976). Issues have been raised concerning the appropriate and useful interrelationships between the model developers, model users, policy analysts, and decisionmakers - often politi- cians. For example, in the field of energy research, Weyant (1980) has recently examined some model applications and analysed organizational conditions for the succesful use of models. He emphasizes the need for improved communication be- tween researchers and decisionmakers for realization of the potential of models as aids to decision making.

0032 2687/82/0000 0000/$02.75 �9 1982 Elsevier Scientific Publishing Company

168

We assert that the problems of organization and of interpreting and communicating model results stem from an underestimation of the epistemological complexity of the use of policy models. Misconceptions about the interpretation of model results seem to originate from a failure to distinguish between different types of models and false analogies between policy models and models in the traditional, more academic disci- plifies. The problems of correctly interpreting model results have been discussed in operations research literature but they seem to be particularly complex and merit further attention in the case of policy models that deal with the long-term development of social systems. For instance, despite the fact that some future-oriented models have been explicitly presented as non-forecasting models, there seems to be a tendency to interpret model outcomes as "forecasts," i.e., relatively unconditional propositions about what will actually happen in the future. In this paper we trace the origin of this problem to the lack of attention to the existence of quite different model concepts. This clarifies the proper use of future-oriented models by making certain distinctions, e.g., between forecasting and such non-forecasting models as "what-if ' and learning models, and, secondly, between different "levels" of results, viz. model outcomes, model inferences and policy-issue oriented interpretations. We will draw on some earlier studies of modeling problems and the development of modeling in various fields. Primarily, however, our discussion is based on an example of an international large-scale computer model recently used for studies of some Swedish strategic energy policy problems which we regard as a typical example of a future-oriented model that is not a forecasting model.

The model, MARKAL, is a technoeconomic energy policy model designed primari- ly for comparative evaluations of new energy technologies. It models the development of the energy system of a region or nation in a long time perspective. MARKAL is a component of an internationally cooperative project within the International Energy Agency, IEA (OECD, Paris). The computer program was developed by research groups at Brookhaven National Laboratory in the United States and the Kernfor- schungsanlage in JtHich, West Germany. The IEA's so-called systems analysis project, which was started 1976 1977, includes not only the development of MARKAL but also the cooperation of experts from a number of countries for the collection of data relating to new energy technologies and the use of MA RK A L by a number of IEA member countries. A joint evaluation of their results was the basis for the development of an IEA energy R & D strategy (IEA, 1980). MA RK A L is a multiperiod linear-pro- gramming model [1]. The time period being studied is divided into nine time steps of 5 years covering the period 1977-2022. The development of the model's energy system is determined by an objective function which may include different optimization criteria, for instances, the minimization of total energy system costs or oil imports.

MARKAL input data include information about current energy systems (existing capacities and remaining lifetime) and about options and constraints for their future development. Results from a MARKAL run include information, for each future time period, about the development of the energy system (for example, the contribution to

169

energy supplies from various kinds of energy sources), the mix between domestic and

imported energy, and the use of existing and new energy technologies (such as wind

power). Figure 1 illustrates the present Swedish energy situation and the types of outcomes that a M A R K A L run yields regarding the contribution from various energy

s o u r c e s .

Abilock & Fishbone's (1979) description of the M A R K A L program emphasizes

that M A R K A L is n o t a forecasting or prediction model and that it is primarily

designed for comparisons of the competitiveness of new energy technologies. But how

does the outcome of such comparisons in the "model world" relate to the real world if

the model produces descriptions of future energy systems which cannot be regarded as

forecasts? We return to this question in section III where we describe some characteris- tics and uses of the M A R K A L model. First, however, we discuss below the concept of

a model, contrasting the "non-forecasting" type of model sometimes used in future-or-

iented policy studies with the traditional model concept of the natural sciences. In the

concluding section, we return to the conceptual and epistemological problems of the

nature and role of mathematical models in future-oriented policy studies and possible

implications for the evaluation and communication of model results.

II The Model Concept

In operations research and management sciences (OR/MS) a model is often defined as "a representation of certain aspects of a real-world system or process." There are many

(P J/YR.)

3000

2000

I000

- PRIMARY ENEROY

HYDRO-ELECTRIC POWER

J .------""""'~OT H E R RENEWABLE

~ A T

NUCLEAR POWER __ NATURAL 6AS---.,7

COAL

OIL AND OIL PRODUCTS

i I ~ I I I 1980 1990 2000 2~0

-T DOMESTIC ENERGY

~ IMPORTED

I - - - - - -

2020

Fig. 1. Different types of primary energy used. A Swedish IEA MARKAL run, 1979.

170

different concepts of models and model classification schemes. Yet, M A R K A L and

MARKAL- type models do not seem to fit the classification schemes commonly used

in the philosophy of science nor can they be viewed as typical OR models. In this

article we will use the expression MARKAL- type models to designate models that are future-oriented

- concerned with policy-actuated development

- deal with phenomena that are both social and technological in nature.

Natural Sciences Mode l s M A R K A L - t y p e Mode l s

In the traditional disciplines a distinction is often made between descriptive and

explanatory models [2]. Fitting a mathematical function to observed data yields a

mathematical model that may be useful as a descriptio n of these data. However, if we have no understanding of the phenomenon observed, the model cannot ultimately be

expected to be reliable if used for forecasting purposes. Yet, there may be cases when

such a model can give fairly accurate forecasts, particularly for short-term forecasting.

When a model is based on some kind of understanding of the observed phenomenon it

is more likely to be useful for forecasting. However, as pointed out by Kaplan (1964), our understanding is often only partial, which is why good explanation does not

necessarily lead to good prediction (pp. 346-349). Additionally, the use of forecasting based on explanation of past developments may be improper where new factors not

discernable from past data influence the system's development.

There are models useful for forecasting which are obtained through the develop-

ment of good explanatory models, as is for instance the case in celestial mechanics.

However, MARKAL-type models do not belong to this category. In fact they are quite

different in kind from the natural sciences models, more or less following the distinc-

tion made by Hermer6n (1977) between mathematicaland theoretical models. Accord- ing to Hermer6n, a theoretical model involves a theory about some process or

phenomenon based on the existence of laws or lawlike statements. This type of model thus involves a certain degree of "understanding" and "explanation." Theoretical

models, when expressed in mathematical form, are also sometimes called mathemati- cal models, but Hermer6n reserves this latter term for non-theoretical models, thus

distinguishing between mathematical and theoretical models. Mathematical, non-

theoretical models might be solely descriptive models or models based on common sense ideas without being expressions of a fully-developed theory. Accordingly, MARKAL-type models would belong to the mathematical, non-theoretical category, whereas the classical models and their successors in the natural sciences are of the theoretical type. In MARKAL, various parts and internal relationships are based on experts' knowledge and elements of theory. But M A R K A L is non-theoretical in the Hermer6n sense, i.e., it is, as a whole, not a theory expressed in mathematical language.

As already mentioned, the M A R K A L builders have stated that it is not a forecast-

171

ing model. This disclaimer is not unique to MARKAL; one finds similar statements as

well in the case of some of the well-known global models. Still, if future-oriented

models are not meant to be used for forecasting purposes, how should they be used?

And can or should any kind of validation be required?

Validation is usually considered as an essential part of modeling work and under-

stood to be a testing of the model by comparisons of model results either with

outcomes of controlled experiments or with historical data. However, experiments

involving complex sociotechnical systems are difficult to design and evaluate. Longi-

tudinal case studies, for example, might only catch a dynamic which at the point of

summary or analysis is either dated or obsolete, and of little use to the action-oriented

policy analyst. No historical tests have been made in the case of M A R K A L and we

would argue that such tests would not have been very useful. A MARKAL-type model

may function well on historical data but be quite useless for studies of future conse-

quences of today's decision alternatives. Future developments cannot be expected to

follow past trends in a policy area such as energy, where government intervention and

policies have considerable influence and sets of actors and technologies, respectively,

are changing.

As traditional natural sciences' or empiricists' procedures for validation are neither

necessary nor sufficient for MARKAL-type models to produce useful results, it is

important to acknowledge them as a separate category of models. In general, histori-

cal validation is hardly relevant for models representing "new" or significantly altered

systems or for future-oriented models that do not purport to predict reality; to require

validation when inappropriate, can be harmful (Greenberger and Richels, 1979;

House and Ball, 1980; Marcuse et al., 1980).

Modellers usually take it for granted that there are legitimate uses of models that are

future-oriented, although not designed for forecasting purposes. These uses are often

not well specified; nevertheless, such models have a number of uses which we discuss

further in connection with our presentation of MARKAL:

- There may be reasons to believe that the model outcomes are of the same order of magnitude as their real-world correspondences. This may be sufficient, for instance,

when only a general frame of reference is needed for some further detailed studies. To

estimate indicators that are of the right order of magnitude has been an explicit

objective of the I IASA global energy modeling project (H~ifele, 1980).

The model may be used for studies of the future impacts of changes in certain input

data, e.g., changes corresponding to the difference between two or more decision

alternatives. Even if a model gives inaccurate forecasts it may sometimes fairly

accurately reflect the effects of such changes on a forecasted variable. A model

designed for this type of use is often called a "what-if' model. The systematic collection of data and the development and use of a model can lead

to an increased understanding of the system being studied. Repeated uses of the model

for different sets of assumptions and input data may highlight what relationships and

data are the most crucial for the outcome, and yield important insights into the

172

problem under study. A model used in this way can be termed a learning model. Unfortunately, to communicate "insights" correctly is much more complicated than

the communication of forecasting results. - The model gives a very "conditional" forecast; it calculates future outcomes under

a set of conditions known to represent a very simplified or even unrealistic picture of

real-world phenomena. What is obtained is a reference case that can be used as a

starting point for further considerations. A specific example is the use of models to calculate reference projections, i.e., future developments that would occur if present

trends were to continue. Assuming that we know a certain trend will change, it may be possible to conclude that t.he development will be above or below the reference

projection. Generally, the use of models to produce reference cases requires a clear

understanding of all the assumptions made as the potential user must be able to integrate the knowledge imbedded in the model and the model outcomes with infor-

mation from other sources.

From OR Models to MARKAL-type Models

The concept of a model in operations research (OR) might be seen as representing a

link between the natural science and the MARKAE- type models. In the 1940s, the objects of inquiry military operations and man-machine industrial systems - were

different from the natural sciences but the paradigm was in one respect the same. A

model of a system could be tested by comparison with empirical data before its use for

prediction and prescriptive recommendations. An important difference between these

types of OR models and natural science models is the possibility of change over time of

man-made systems; validation becomes time-dependent as the predictive relevance

erodes with change. As a consequence, the model becomes more of a tool of investiga-

tion and less of an end in itself.

With the evolution of operations research during the last decades into new areas for

inquiry, strategic problems, and complex social systems, the situation has changed. Validation of models on the basis of comparisons between the behavior of the model

and the real-world system being modeled is often not feasible. This development has necessitated a change in quality criteria from results to process-oriented evaluation as

emphasized by Majone (1980a) in his examination of the paradigmatic evolution

of operations research, systems analysis, and policy analysis. The introduction of new quality control terms and activities, such as model evaluation (Gass, 1977), model analysis (Greenberger, 1980; Greenberger et al., 1976), and model assessment, (Gass,

1980; Greenberger and Richels, 1979), and an increasing attention to problems of

correctly communicating model results (Brewer, 1978; Majone, 1980b; Meltsner, 1980; Roth et al., 1979; Sweeney and Weyant, 1979) can be interpreted as an expression of this paradigmatic shift. This development has been more discernible in the United States and in the energy field than in other countries and fields of application.

173

Because of the inherent difficulties in validating models of systems not yet in

existence, validation in its traditional sense has shifted towards analysis of model assumptions and testable parts of the model as explored by Van Horn (1971) in the case of simulation models. But the term "validation" has also been given new meanings by the choice of new bases for comparison. For instance, model behavior and validity may be compared with first, other models; or second, general understanding, i.e., with the mental image of the assessor (Goldman and Gruhl, 1980:114). Ziegler's (1975) approach corresponds to the first as he defines validation of a model as a test of agreement with a more elaborate model of the same system. He shows that predictive validity requires homomorphism. Greenberger has proposed new forms of validation that basically correspond to the second, and uses such criteria as "face judgement, sensitivity runs, and linkage checks" (Greenberger, 1980; Greenberger et al., 1976). Various experts, for instance, may contribute their criticisms of and revisions to the appropriate levels and components of the model, to its important hypotheses and principles. By methods such as sensitivity analysis, one can examine what parts of a model and what input data are decisive for certain types of results (Thissen, 1978). But in the case of MARKAL-type models, which deal with the long-term development of sociotechnical systems, not even these weaker types of validation allow any overall validation of the model. They have to be limited to specific model uses. Applying Majone's process-oriented paradigm would then mean that quality criteria should encompass the entire process from model development and use to the communication of policy-relevant information.

In the table below we summarize a simplified description of some of the differences between theoretical and mathematical models when used to explore future develop- ments. As regards mathematical models, we only include MARKAL-type models which represent a somewhat extreme case.

Quality Criteria Uses

Theoretical Traditional model Forecasts models validation

MARKAL-type models

"Validations" of specific model uses, but emphasis on the entire process leading to policy- -relevant information

Forecasts of "order of magnitude"; What-if questions;

Learning; Reference cases.

Thus, in some branches of applied systems analysis and policy analysis, a new model concept, new validation concepts, and new paradigms have emerged which are dis- tinctly different from both traditional O R / M S models and the theoretical models of the traditional disciplines. Yet the risk of false analogies between different model

174

concepts remains. When modeling becomes popular in a new field of application, modellers of tencome directly from some university discipline, indoctrinated with the conventional model concept of their discipline. For instance, philosophy of science

literature is concerned with descriptive, explanatory and forecasting models but has not given much attention to the normative models of operations research and man- agement science, nor does it acknowledge the existence of future-oriented, non-fore- casting models. A diminishing of the risk of false model analogies seems to require the recognition of a fuller range of model concepts in all academic disciplines and a greater consensus regarding the meaning and role of model validation.

III The M A R K A L Model

Numerous energy models developed during the 1970s addressed a variety of issues such as energy pricing, the development of particular energy technologies, energy demand, and energy economy interactions. Manne et al. (1979) survey the develop- ments in the U.S. Our choice of example of energy model, MARKAL, partly results from one of the authors' role in evaluating the Swedish applications made as part of the IEA project. Furthermore, the problems of interpreting results from MARKAL seem similar to those debated in the use of models in other fields, e.g., in connection with the world models.

MARKAL was designed for studies of a range of problems concerning new energy technologies. A distinction is made between supply and demand technologies. Ex- amples of supply technologies would be wind power plants, conversion of coal to synthetic fuels or gas, methanol production Hom biomass, and solar cells. Demand technologies are, for instance, various types of automobile engines and heat pumps.

The model calculates a development of the energy system that meets a demand for useful energy that is exogenously determined for different time periods and uses within different end-use sectors of the economy.

A MARKAL run optimizes the energy system over a 45-year period. The model assumes perfect foresight; decisions are implicitly assumed as taken with complete information, e.g., about future energy prices or technologies available in the future. This is, of course, quite an unrealistic assumption. Nevertheless, given this assump- tion, it is useful in strategic planning to know, as a point of reference, how a given or specified energy system might develop. More generally, MARKAL can calculate how the energy system might develop under a large set of different assumptions, some of which are well known to be unrealistic.

There is no '~ description of the energy system in the model. Condi- tions that depend on geographical factors, such as transportation costs and energy transmission losses, have to be taken into account when the input data are assigned values, usually by using estimates of mean values. Input data quite generally represent mean values, with variations around the mean values usually disregarded. Conse- quently, when such variations may have important effects, such effects have to be

175

considered in the evaluation of the model outcomes. Inferences from model runs

therefore need to be based upon a synthesis of the model outcomes and the speculated

effects on these outcomes from da ta /model oversimplification.

The IEA's purpose in developing and using M A R K A L was to obtain comparable data about the usefulness of new technologies for reducing members ' oil imports

dependencies. Thus, there was an emphasis on consistency in the member countries'

uses of MARKAL. Less attention was paid to the possible adaptation of M A R K A L to

constraints and objectives stemming from the energy policies of a specific country. For

the IEA M A R K A L simulations, two objectives dominated the long-term develop-

ment of the Swedish energy system, namely the minimization of the total energy

system costs and the limitation of oil imports. But there are other objectives of

importance in Swedish energy policy, for instance - limiting the dependence on all energy imports;

- environmental considerations; and

- restrictions on the use of nuclear energy In some new Swedish M A R K A L runs, additional objectives were taken into

account (Schwarz and Lekteus, 1980). Thanks to the great flexibility of the computer

program, this change to the model could be made without extensive reprogramming.

Figure 2 illustrates the model outcomes as regards the use of different energy

(PJ/YR.)

4000

3000

2000

1000

-PRIMARY ENERGY AND CONSERVATION

/ HYDRO" ELECTRIC POWER

f

NUCLEAR POWER

OIL AND OIL PRODUCTS

I t 1990

. ~ ~ ~ ~ CONSERVATION

~ ~ " CONSERVATION /

OTHER RENEWABLE DOMESTIC ENERGY

IMPORTED ENERGY

- L i I i - - - - 2010 2020

P ~

COAL S_y NT..HE_TIC_F.U_ELS_A_ND _GAS_7 - _ _.

I I 1980 2000

Fig. 2. Primary energy and the effect of conservation in the industrial and residential sectors. A M A R KAL

run in Schwarz and Lekteus (1980).

176

carriers for a case where nuclear power was assumed to be phased out, corresponding

to one of the policy alternatives in the Swedish referendum on nuclear policy in

March 1980. This alternative is defined as a continuation of the present twelve nuclear

power reactors program for the lifetimes of these plants (until approximately 2010). One important type of result f rom M A R K A L projections is whether or not a new

energy technology is used at all, as well as the amount of its use and point of

introduction. Still, the viability of various energy technologies depends on the mix of

energy carriers selected. Comparing Figs. 1 and 2, we see that there is a considerable

difference in the use of energy carriers between the IEA runs and the new computa-

tions. As could be expected, the resulting use of energy technologies was also different,

although some results were not very sensitive to the changes between the IEA and the

new runs. M A R K A L was designed as a "what-if" model for new energy technologies, that is,

for comparisons of different technologies and for estimates of what difference it would

make ifa certain technology was or was not available at some point in the future. This approach is predicated upon the expectation that the marginal impact of such changes

in input data on the model outcomes may be reasonably analogous to the correspond- ing real-world impact even if the model is a very incomplete representation of the

system being modelled. Such analogies cannot generally be warranted. There are, for instance, some new energy technologies whose viability to a large extent depend on

what energy sources will be used in the future which, in turn, can be considerably

influenced by national policies and uncertain international developments. The use of

M A R K A L for this type of what-if problems thus requires careful assessments of the extent to which the model captures essential factors of change, such as new policies

and sensitivity analyses for uncertainties that can have a major impact on such features of the development of the energy system. These, in turn, influence the competitiveness

and the market penetration levels of the new technologies. The necessity of these

requirements is illustrated by the difference in results regarding the introduction of

energy technologies between the IEA and the new M A R K A L runs, where certain

restrictions due to environmental concerns and nuclear power limitations were intro-

duced. To facilitate discussing the use of MARKAL- type models, it seems to be useful to

distinguish between different "levels" of results from studies involving the use of

models. The importat ion of natural gas to Sweden on a large scale has been a crucial policy issue for the Swedish government. In the M A R K A L run illustrated in Fig. 2, natural gas is computed to be imported but to a very limited extent. Similar results were obtained from all the other M A R K A L runs for Sweden. A closer examination of input data and the structure of the representation of the natural gas issue in the model revealed the need for interpreting the model outcomes in light of over-simplified assumptions and uncertain input data. Investment costs, for instance, were given as averages, yielding, because of scale effects, an underestimation of costs in the case of very limited use of natural gas. Additionally, it was found that the assumption made

177

about natural gas import prices in 15 20 years time had a very decisive effect on

outcomes. This price may depend on the number of contractors and a variety of

political factors which are beyond the scope of the model. The conclusion drawn was

that there are other energy system developments than the one represented by the model outcomes that are likely to be better and more feasible, for example, no importat ion of natural gas or more extensive use of natural gas with emphasis on

flexibility in the design of the natural gas system (Schwarz and Lekteus, 1980).

Drawing from this example, we may distinguish between three categories or levels

of results obtainable through the use of non-forecasting models:

1. model outcomes; 2. inferences that can be made on the basis of some kinds of correction for deficiencies

in input data and model structure; and 3. interpretations of results that can be related to a policy issue and are based on a

synthesis of type 2 results with information outside the scope of the model.

The distinction between different levels of results seems useful to facilitate discussions

about the use of future-oriented policy models and the problems of interpreting and

communicating model results.

IV On the Use of non-Forecasting Models

As mentioned earlier, future-oriented models are not necessarily intended to be used

for forecasting or prediction purposes. They may be used as "what-if" models or, more

generally, to produce reference developments or yield insights into the problems being studied. In the field of global modeling, for instance, Meadows et al. (1972: 142)

emphasize that the outputs from their model are not predictions. Leontief (1977) considers his model primarily as a"p lanning device" and Mesarovic and Pestel (1974)

see their model as a tool to experiment with policies (see also Hughes, 1976).

Models used for sensitivity analysis belong to the model category that we have

called "what-if" models. Keyfitz (1979: 198) points out in his comparison of the world

models that "the main use of the models is not for forecasting but for sensitivity

analysis." We agree with Keyfitz about sensitivity analysis being one main use for non-forecasting models, yet this use has its limitations and pitfalls. Sensitivity analysis

is usually understood as an examinat ion of the effects on model outcomes of certain

changes in input data. However, there is no general correspondence between these outcomes in "the model world" and the real-world unless the model is structurally

valid, i.e., is homomorphic with the modeled system (Ziegler, 1975). In non-forecast- ing models, this structural validity is typically only partial. In MARKAL, for exam- ple, sensitivity analysis regarding what difference it makes whether some new energy technology is available or not may give misleading results if the competiveness of the technology depends on factors which have not been incorporated in the model, such as environmental or nuclear power policies. Another type of pitfall in the use of models

178

for sensitivity analysis has been discussed by Quade (1980: 33) and Holling (1978:103), who point out that it may be misleading to test the sensitivity to a series of uncertain parameters by changing them one by one, as simultaneous variation of the parameters may give quite different results. Holling cites a study of the Meadows' world model by

Scolnic (1973) that clearly illustrates this phenomenon. There seems to be a growing consensus among those who have been engaged in

comparative studies of different policy models that the models can be useful, not because of the specific figures for future developments they produce, but because of the more general insights they can yield. Ira model directly produces reliable forecasts or optimal policies, then interpreting the model outcomes and communicating them to decisionmakers is a comparatively straightforward process. This is not the case for what-if models or when models only yield insights, that is, function as learning models. In these cases, the model user either has to communicate what has been learnt in a way which allows others (policy analysts or policymakers) to bring this knowledge to bear on policy issues. Or the model user himself has to translate the knowledge gained into policy-relevant information. For the use of non-forecasting models for policy prob- lems, we can distinguish between the following three possibilities:

(i) The model user (who may or may not be the same person as the model developer) presents model outcomes and inferences that can be made on the basis of some kind of correction for deficiencies in input data and model structure, i.e., the model user limits his presentation of results to level (1) and (2) in our categorization of model results (c.f.

section III). (ii) The model user makes interpretations of results that are directly related to policy

issues and based on a synthesis between model inferences and information outside the

scope of the model, i.e., the model user focuses on level (3) results. (iii) A specific policy problem is the point of departure and is analyzed by the use of

a number of different models. In case (i), the use of the results for policymaking is problematic because the

policymaker may find difficulties in identifying and assessing the impacts of factors which are potentially important for policy but not adequately treated in the model. Responses to this problem in the case of non-forecasting models include an emphasis on making the models simpler or more transparent, improving the documentation of the model, and model analysis that can illuminate a model's important assumptions and its underlying structure. This is contra distinctive to the route normally chosen in the case of forecasting models where inadequacies are met by further developing the models. Case (ii) poses less requirements for model documentation and model trans- parency but involves the risk of yielding misleading policy recommendations when the model user is an expert in the areas covered by the type of model used but not necessarily an expert in the particular policy issue. A response to this problem can be to establish two-way communication between the model user and the policymaker throughout the entire modeling and interpretation processes. Another approach to handling the limitations of expertise is to evaluate comparatively several models

179

addressing the same policy area. This approach leads us to case (iii).

As Richardson (1978) and others make clear, the structure of a model of a complex

social system is mainly determined by the modeller 's world view and academic

background, not necessarily by what hypothetically would be considered the most valid structure for a specific range of policy issues. A possible remedy is the simultane-

ous use of several models. This would be in line with Dror 's (1975) suggestion for

addressing complex, important policy problems by "the simultaneous use of multiple

and diverse assumptions, models, languages and techniques". Starting from a specific

policy problem and using several models, which possibly are different in structure and

emphasis, one might distill information relevant to the policy problem. An organiza- tional form for this approach, "a modeling forum," has been suggested by Greenberger

(1980) and put into operation in the United States in the energy policy field (Sweeney

and Weyant, 1979). We consider this as an example of case (iii) above. An illustrative

example of a modeling forum study is the Energy Modeling Forum report on the

development of oil prices (EMF, 1982). For a set of different scenarios, ten models

were run by experts. Workshops were arranged to discuss the causes behind the

various outcomes and investigate what oil policy conclusions could be drawn in view

of existing uncertainties and limitations of the models. Further examples in the

direction of a type (iii) approach are the I IASA series of conferences on global

modeling and various comparative studies, such as the world models studies under-

taken at the Sussex Science Policy Research Unit (Clark and Cole, 1975; Cole, 1977). In summary, the interpretation of model results is a complex undertaking in the case

of future-oriented policy models of sociotechnical systems. To make knowledge and

experiences of model evaluation issues more cumulative and easier to communicate

across disciplinary boundaries and between different fields of application, we need to

distinguish more clearly between various types of models, such as theoretical and

mathematical models, and between forecasting and such non-forecasting models as

what-if and learning models. Possibly such a development is not feasible unless a fuller range of model concepts is taught in all academic disciplines and more of a consensus

develops regarding the meaning and role of model validation, for various purposes.

Acknowledgement

We are indebted to Professor Ken Bowen (Department of Mathematics, University of

London), Professor G6ran Hermer6n (Department of Philosophy, Lund university),

and two anonymous referees for comments on an earlier draft. The authors are, of

course, solely responsible for all assertions and any remaining errors.

Notes

l A short description of the computer program has been published by Fishbone (1979) and a more complete "user's guide" by Abilock & Fishbone (1979).

2 This and the following paragraph are partly based on a section on mathematical models in Schwarz et al. (1980) Met hods in Futures Studies - Problems and Appl icat ions .

180

References

Abilock, H. and L. Fishbone (1979). "User's Guide for MARKAL" (BNL Version), BNL-27075, IEA Energy Systems Analysis Project, Brookhaven National Laboratory, Upton, New York.

Brewer, G. D. (1978). "Operational social systems modeling: pitfalls and prospectives," Poli~ 3' Sciences 10: 157-169.

Clark, J. and S. Cole (1975). Global Simulation Models A Comparative Stuctl'. London: John Wiley & Sons.

Cole, S. (1977). Global Models and the htternational Economic Order. Oxford; Pergamon Press, 1977. Dror, Y. (1975). "Some features of a meta-model of policy studies," Policy Studies Journal 3:247 255. Energy Modeling Forum (1982). Worm Oil, EM F Report 6. Stanford, CA: Stanford University. Fishbone, L. (1979). "MARKAE, A Multiperiod Linear-Programming Model for Energy Systems Analy-

sis" (BNL Version), BNL-26390, Brookhaven National Laboratory, Upton, New York. Gass, S. I. (1977). "Evaluation of complex models," Computers and Operations Research 4: 27-35. Gass, S. l. (ed.) (1980). Validation and Assessment Issues gf Energy Models. U.S. Department of Com-

merce. Goldman, N. L. and J. Gruhl (1980). "Assessing the ICF coal and electric utilities model," in S. I. Gass (ed.),

Validation and Assessment Issues o f Energy Models. Greenberger, M. (1980). "A way of thinking about model analysis," lnte~Jaces 10 (2): 91-96. Greenberger, M. and R. Richels (1979). "Assessing energy policy models: current state and future direc-

tions," Annual Review of Energy 4: 467-500. Greenberger, M., M, Crenson and B. Crissey (1976). Models in the Polic:v Process: Public" Decision Making

in the Computer Era. New York: Russell Sage Foundation. Hermer6n, G. (1977). "Models in forecasting and analysis," in C.-G. J ennergren, S. Schwarz and O. Alvfeld

(eds.), Trends in Planning. Chichester, England: Wiley. Holling, C. S. (ed.) (1978). Adaptive Environmental Assessnwnt and Management. Chichester, England:

Wiley. House, P. W. and R. H. Ball (1980). "Validation a modern day snipe hunt? Conceptual difficulties of

validating models," in S. ]. Gass (ed.), Validation and Assessnwnt Issues o/" Energy Models. Hughes, B. B. (1976). "Survey of the Mesarovic-Pestel world model project," in H. Bossel, S. Kiaczko and

N. M~3tler (eds.), Systems Theory in the Social Sciences. Basel and Stuttgart: Birkh~iuser. H~ifele, W. (1980). "IIASA's World Regional Energy Modelling," Futures (Feb.) pp. 18 34. International Energy Agency (1980). "A Group Strategy for Energy Research, Development and

Demonstration. OECD, Paris. Kaplan, A. (1964). The Co;.Tduct oflnquiJ3'. San Francisco: Chandler. Keyfitz, N. (1979). "Understanding World Models," Behavioral Science 24:190 199. Lee, D. (1973). "Requiem for large-scale models," AlP Journal, May, pp. 163 178. Leontief, W. et al. (1977). The Future o f the Worm Economy. United Nations, New York. Majone, G. (1980a). "'Applied Systems Analysis: A Genetic Approach," WP-80-61, IIASA, Laxenburg,

Austria, April 1980a. Majone, G. and E. Quade (eds.), Pitfalls o f Analysis. Chichester, England: John Wiley & Sons for IIASA. Majone, G. (1980b). "An anatomy of pitfalls," in G. Majone and E. Quade (eds.), Pit/~tlls o f Analysis. M anne, A., R. Richels and J. Weyant (1979). "Energy policy modeling: a survey," Operations Research 27,

( I ) : l 36. Marcuse, W., F. T. Sparrow and D. A. Pilati (1980). "Validation issues: a view from the trenches," S. I. Gass

(ed.), Validation and Assessment Issues o f Energy Models. Meadows, D., et al. (1972). The Limits to Growth. New York: Universe Books. Meltsner, A. J. (1980). "Don't slight communication: some problems of analytical practice," in A. Majone

and G. Quade (eds.), Pitfalls o f Analysis. Mesarovic, M. and E. Pestel (1974). Mankindat the Turning Point. New York: E. P. Dutton. Quade, E. S. (1980) "Pitfalls in formulation and modeling," Chapter 3 in G. Majone and E. Quade (eds.),

Pitfalls" o f Analysis. Richardson, J. (1978). "Global modelling," Futures, Dec.

181

Roth, P. F., S. 1. Gass, and A. J. Lemoine (1979). "Some considerations for improving federal modelling," in B. P. Zeigler, M. S, Elzas, G. J. Klir and T. 1. Oren (eds.), Methodology in Systems Modeling and Simulation. Amsterdam, New York, Oxford: North-Holland.

Sailor, V. (ed.) (1979). "Technology Review Report", IEA Energy Systems Analysis Project, BNL-27074, Brookbaven National Laboratory, Upton, New York.

Sanders, R. (1973). The Politics o f Defence Analysis, New York: Dunellen. Schwarz, B. and I. Lekteus (1980). "'Energy Systems, Multiple Objectives, and New Technologies," (in

Swedish), SP 1980: 1, Economic Research Institute, Stockholm, Sweden. Schwarz, B., U. Svedin and B. Wittrock (1982). Methods in Futures Studies." Problems and Applications.

Boulder, CO: Westview. Scolnik, H. D. (1973). "On a Methodological Criticism of the Meadows World 3 Model,"Tech. Rep. Dept.

Math., Fundacion Bariloche, Argentina. Sweeney, J. and J. Weyant (1979). "The energy modeling forum: past, present and future," Journal o f

Business Administration, Special Issue on Resource Policy Analysis, 1979. Thissen, W. (1978). "Investigations into the World 3 model: lessons for understanding complicated

models," IEEE Transaction on SvstenTs, Man and Qvbernetics. Vol. SMC-8, No. 3 (March). Van Horn, R. (1971). "Validation of simulation results," Management Science, 17 (3). Weyant, J. (1980). "Quantitative models in energy policy," Policy A nalysis: 211-234. Ziegler, B. (1975). Theory o f Modelling and Simulation. New York: John Wiley & Sons.