Research Funding as an Investment: Can We Measure the Returns?

Research Funding as an Investment: CanWe Measure the Returns?

April 1986

NTIS order #PB86-218278

Recommended Citation:Research Funding as an Investment: Can We Measure the Returns?-A Technical Memoran-dum (Washington, DC: U.S. Congress, Office of Technology Assessment, OTA-TM-SET-36, April 1986).

Library of Congress Catalog Card Number 86-600525

For sale by the Superintendent of DocumentsU.S. Government Printing Office, Washington, DC 20402

Foreword

The mounting intensity of global competition in the 1980s underscores the criticalrole played by the basic research enterprise of the United States. Basic research is thebackbone of much of the technological development that has provided not only oureconomic prosperity, but improvements in health, a strong national defense, and excit-ing and fundamental advances in knowledge.

Congress is faced with difficult decisions regarding funding for research. The HouseScience Policy Task Force asked OTA to provide information on the extent to whichdecisionmaking would be improved through the use of quantitative mechanisms associ-ated with the concept of investment. If investing in science is similar to investment inthe financial sense, can the returns be meaningfully predicted and measured? Can reason-able investment criteria be devised? OTA concluded that while there are some quan-titative techniques that may be of use to Congress in evaluating specific areas of research,basic science is not amenable to the type of economic analysis that might be used forapplied research or product development. OTA also concluded that even in the busi-ness community, decisions about research are much more the result of open communi-cation followed by judgment than the result of quantification.

Much of the vitality of the American research system lies in its complex and pluralis-tic nature. Scientists, citizens, administrators, and Members of Congress all play vari-ous roles leading to final decisions on funding. While there may be ways to improvethe overall process, reliance on economic quantitative methods is not promising. Ex-pert analysis, openness, experience, and considered judgment are better tools.

Director

.,,///

Workshop Participants

Harvey AverchNational Science FoundationRadford Byerly, Jr.U.S. CongressHouse Committee on Science and

TechnologyDaryl ChubinGeorgia Institute of Technology

Henry HertzfeldConsultantJohn HolmfeldU.S. CongressHouse Committee on Science and

TechnologyDavid MoweryCarnegie-Mellon Institute

Reviewers and Other Contributors

William Alumkal Zvi GrilichesE.I, du Pont de Nemours & Co. Harvard University

Nicholas AshfordMassachusetts Institute of

Technology

Alden BeanLehigh University

Edwin L. BehrensProctor & Gamble

Harvey BrooksHarvard University

Charles BuffalanoDefense Advanced Research

Projects Agency

H. Roberts CowardCenter for Research Planning

Susan CozzensNational Science Foundation

Gordon EnkInternational Paper Co.

Daniel FinkConsultant

James FletcherConsultant

J. Jeffrey FranklinCenter for Research

Howard GobsteinGeneral Accounting

Planning

Office

Walter HahnGeorge Washington University

Christopher HillCongressional Research Service

Noel HinnersNational Aeronautics and Space

Administration

Allan R. HoffmanNational Academy of Sciences

Benjamin HubermanConsultants International

Bill KirschUniversity City Science Center

Genevieve KnezoCongressional Research Service

Charles F. LarsonIndustrial Research Institute

Fred LeavittDow Chemical, USA

John M. LogsdonGeorge Washington University

Alexander MacLachlanE.I. du Pont de Nemours & Co,

L.M. MagnerE.I. du Pont de Nemours & Co.

Edwin MansfieldUniversity of Pennsylvania

J. David RoessnerGeorgia Institute of TechnologyNestor TerleckyjCenter for Socio-Economic

AnalysesNational Planning Association

Bruce MazlishMassachusetts

TechnologyInstitute of

Norman MetzgerNational Academy of Sciences

Robert P. MorganWashington University

Nathan RosenbergStanford University

Mark RossUniversity of Michigan

Charles F. SabelMassachusetts Institute of

Technology

D.M. Von SchriltzE.I. du Pont de Nemours & Co.

Ora SmithOffice of Science and Technology

Policy

Elly ThomasNational Science Foundation

John N. WarfieldGeorge Mason University

Robert WeaverPennsylvania State University

Philip WebreCongressional Budget Office

iv

Research Funding as an Investment: Can We Measure the Returns?OTA Project Staff

John Andelin, Assistant Director, OTAScience, Information and Natural Resources Division

Nancy Carson NaismithScience, Education, and Transportation Program Manager

Project Staff

Eugene Frankel, Project Director (After September 1985)

Barry J. Holt, Project Director (Through September 1985)

Kathi Hanna Mesirow, Analyst

Lisa Heinz, Analyst

Mia Zuckerkandel, Research Assistant

Marsha Fenn, Administrative Assistant

Betty Jo Tatum, Secretary

Chris Clary, Secretary

Gala Adams, Clerical Assistant

Contractors

Evan Berman Daryl Chubin Kevin Finneran

Henry Hertzfeld David Mowery J. David Roessner

Other OTA Contributors

John Alic Philip P. Chandler II

Marcel C. LaFollette Martha Harris

Contents

PageCHAPTER 1: Executive Summary: Overview

and Findings . . . . . . . . . . . . . . . . . . . . . . . . . 3Economic Returns . . . . . . . . . . . . . . . . . . . . . . . . 3Bibliometrics and Science Indicators . . . . . . . . 6Research Decisionmaking in Industry and

Government . . . . . . . . . . . . . . . . . . . . . . . . . 7Summary? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

CHAPTER 2: Measuring the Economic Returns:Progress and Problems . . . . . . . . . . . . . . . . 13

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Econometric Studies at the National Level . . . 13

Private R&D Investment . . . . . . . . . . . . . . . . 13The Returns, or Lack Thereof, to Federally

Funded R&D in Specific Industries. . . . 14Industry Case Studies: Agriculture, Aviation,

and Health . . . . . . . . . . . . . . . . . . . . . . . . . . 16Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Aviation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Spinoffs and Spillovers: NASA. . . . . . . . . . . . . 22Macroeconomic Studies . . . . . . . . . . . . . . . . . 22Macroeconomic Studies. . . . . . . . . . . . . . . . . . 23Patent Analysis . . . . . . . . . . . . . . . . . . . . . . . . 23

Implications for Using Economic InvestmentModels To Guide Federal R&DDecisionmaking . . . . . . . . . . . . . . . . . . . . . . 24

CHAPTER 3: Noneconomic QuantitativeMeasures: The “Output” of Science . . . . . 29

Bibliometrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29The First Generation of Bibliometrics

(1961-74) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30The Second Generation (1975-85) . . . . . . . . . 31Other Important Teams . . . . . . . . . . . . . . . . . 36

The Use of Bibliometrics To EvaluateResearch at the National Institutesof Health . . . . . . . . . . . . . . 37

NIH Databases . . . . . . . . . . . . . . . . . . . . . . . . . 37Bibliometric Studies at NIH. . . . . . . . . . . . . 38The Utility of Bibliometrics for Research

Decsionmaking . . . . . . . . . . . . . . . . . . . . . . 40Science Indicators . . . . . . . . . . . . . . . . . . . . . . . . 44

CHAPTER 4: Research Decisionmaking inIndustry: The Limits to QuantitativeMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Review and Evaluation of OngoingResearch Activities. . . . . . . . . . . . . . . . . . . . 47

Microvconomic Models . . . . . . . . . . . . . . . . . . 48Business Opportunity Techniques . . . . . . . . . 49

R&D Project Selection . . . . . . . . . . . . . . . . . . 49

PageUse of R&D Project Selection Models

in Industry . . . . . . . . . . . . . . . . . . . 51Reasons for Levels and Patterns of

Observed Use . . . . . . . . . . . . . . . . . . . . 52Strategic Planning and Resource Allocation. 54

CHAPTER 5: Government Decisionmaking. . . . 61The R&D Budgetary Process . . . . . . . . . . . . . . . 62project Selection . . . . . . . . . . . . . . . . . . . . . . . 66Research Program Evaluation . . . . . . 68Forecasting and Strategic Planning. . . . . . . . . . 69

Planning for Innovation in Japan . . . . . . . . 69

List of TablesTable No. Page1. Federal Obligations for Research and -

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

Development by Character of Work andR&D Plant: Fiscal Years 1984-85 . . . . . . . . . . 5Quantitative Methods Used To EvaluateR&D Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Expenditures by the Public Sector onResearch and Extension Oriented toImproved Agricultural ProductionTechnology 1890-1970 . . . . . . . . . . . . . . . . 17Econometric Studies of ProductivityReturn to Agricultural Research in theUnited States . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Economic Costs of Disease:Differential Estimates . . . . . . . . . . . . . . . . . . . . 21Main Problems With the Various PartialIndicators of Scientific Progress and Detailsof How Their Effects May Be Minimized. . . 34Output Indicators for Optical Telescopes—A Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Experimental High-Energy Physics,1977-80 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Assessments of Main Proton Acceleratorsin Terms of “Discoveries” and “ProvidingMore Precise Measurements” . . . . . . . . 35Percent Change in Sample of ContinuingClusters, 1970-73 . . . . . . . . . . . . . . . . 41Federal Obligations for Research andDevelopment by Character of Work andR&D Plant: Fiscal Years 1956-85 . . . .

List of FiguresFigure No.

1.2.

3.4,

Annual R&D Investment, 1945-82 .,.Development of a Specialty Cluster,1970-73 . . . . . . . . . . . . . . . . . . . . . . . . . .Formulation of the President’s BudgetThe Congressional Budget Process.

. . . . 02

Page. . . . . 19

. . . 4262

. . . . 64

vi

Chapter 1

Executive Summary:Overview and Findings

Chapter 1

Executive Summary:Overview and Findings

The assumption that federally funded scientificresearch leads to economic benefits for the coun-try has been fundamental to government sciencepolicy since the end of World War II. Analystshave abandoned the linear model that sees a sim-ple progression from basic research to appliedresearch to product development, but they stillbelieve that scientific research plays a vital rolein technological progress and consequently in eco-nomic growth. Economic returns, however, areneither the sole nor the primary purpose forFederal research spending. The advancement ofknowledge and specific mission agency goals suchas national security, public health, and the explo-ration of space are all essential parts of the ra-tionale for Federal research spending.

Several trends have combined in recent yearsto make some policymakers more interested ineconomic and other quantifiable measures of re-search success and benefits. Technology is becom-ing an essential component of economic competi-tiveness; Federal budget constraints are forcinglawmakers to reevaluate spending and to look forways to compare the value of widely divergentgovernment programs; quantification of programsuccess offers the hope for an objective measurethat could simplify politically contentious deci-sions about increasingly esoteric and complexscientific research. One approach to simplifyingresearch evaluation is to view Federal research

spending as an investment that should producea measurable economic return.

The Task Force on Science Policy of the HouseCommittee on Science and Technology has raisedthe issue of whether the metaphor of researchfunding as an investment can be used as a practi-cal aid to Federal research decisionmaking. “CanFederal funding for science be viewed as an in-vestment and be measured in a way comparableto other forms of economic investment?” theCommittee asked in its Report on a Study ofScience Policy. Specifically, the Committee askedO-J-A to study “the models and other analyticaltools developed by economists to judge capital in-vestments, and the applicability and use of thesemodels and tools to government funding of scien-tific research. ”

To carry out this study, OTA conducted a com-prehensive search of the literature on the economicreturns to investment in scientific research, metwith numerous economists and public policy ana-lysts who have studied this issue, conducted in-terviews with research decisionmakers in indus-try and in government, and carried out in-depthstudies of the quantitative methods available toevaluate the progress of scientific research. Thistechnical memorandum presents the findings ofthat investigation.

ECONOMIC RETURNS

Economists have shown a strong positive corre- eral R&D expenditures, except for some appliedlation between research and development (R&D) research programs in agriculture, aeronautics, andspending and economic growth. They have esti- energy designed to improve industrial productiv-mated private returns in excess of 20 percent per ity. These findings are discussed at length in chap-year and social returns in excess of 40 percent on ter 2.private sector R&D expenditures. They have notbeen able to show comparable returns, and at The economists who have carried out thesetimes been unable to show any returns, on Fed- studies point out a number of reasons why eco-

3

4

nomic return on investment calculations may beinappropriate for evaluating government R&D ex-penditures. First, the return-on-R&D-investmentstudies carried out to date measure an average re-turn on a total previous investment. They givelittle guidance as to the marginal return that canbe expected from the next incremental investmentin R&D, which is the decision that policymakersmust make. Second, most government expendi-tures, including R&D expenditures, are for so-called “public goods” whose market value is, bydefinition, extremely difficult to measure in eco-nomic terms. Third, despite the success of retro-spective studies, there are no reliable formulae torelate future R&D expenditures to productivityimprovements or other economic benefits. Predic-tions of future returns on investment cannot bemade without such relationships.

Financial counselors and economists have de-veloped techniques for selecting investments insituations involving risk and uncertainty. Thesetechniques include capital investment methodol-ogies, portfolio analysis, and financial investment’models. All of these techniques have heuristicproperties that could guide investment in researchand development. For example, spreading the riskamong a number of projects is one response econ-omists sometimes recommend in cases involvinggreat uncertainty. However, the formal modelsthemselves are not especially helpful to researchdecisionmaking. They assume that the decision-maker can estimate in dollar values the benefitsfrom potential investments and know or estimatethe probability of achieving those benefits. Nei-ther of these assumptions is applicable to gover-nment-funded research, except in special cases. Theprincipal benefit of research, especially basic re-search, is new and often unexpected knowledge,which cannot be assigned a direct economic value.

Investment models also assume that the bene-fits of the investment return to, or are appropri-able by, the investor. New knowledge-by con-trast—is available to anyone to use. This is oneof the reasons basic research is considered a pub-lic good, requiring government support.

Research leads to productivity improvementsand economic growth primarily through techno-

logical innovation. However, the relationship be-tween research and innovation can be long-term,indirect, and unpredictable. Studies of technologi-cal innovations have shown them to depend onresearch results that are decades old and often inseemingly unrelated fields. Moreover, the trans-formation of research into economically success”ful innovation depends on factors in the economythat are completely outside the research process.These factors include the climate for investment;government tax, regulatory, and patent policy;the degree of competitiveness and entrepreneurial-ism in industry; the state of the capital markets;foreign competition; and wages, unionization, andother characteristics of the work force. A highlysuccessful basic research effort may never gener-ate technological innovation or economic payoffif other factors in the economy are not conduciveto technological change.

Some observer: argue that if economic returnsare to be the primary measure of our research ef-fort, we should focus our attention, as a Nation,on the factors that link science to technology andinnovation. The United States spends less of itsR&D budget than West Germany, France, Eng-land, or Japan on research related directly to in-dustrial productivity. Efforts to improve that sit-uation could include an increased emphasis ontechnology transfer, increased support for genericresearch related to industrial needs, and adapta-tion of more focused forecasting and planning forindustry-related R&D. (See ch. 5.)

Applied research, whose goal is the solution ofpractical problems, can be more closely associ-ated with economic activity. However, most ofthe applied research in the-Federal Governmentis carried out by agencies whose mission objec-tives—defense, health, space—are not readilyquantifiable in dollar terms. Table 1 shows theestimated 1985 Federal basic and applied researchbudgets by agency. As can be seen, in applied re-search the Departments of Defense (DOD) andHealth and Human Services (DHHS) were the twolargest contributors, with the Department ofEnergy (DOE) and the National Aeronautics andSpace Administration (NASA) third and fourth.All of the DOD and DHHS applied research re-

5

Table l.— Federal Obligations for Research and Development by Character of Work and R&D Plant:Fiscal Years 1984-85 (thousands of dollars)

Research

Total R&D and Basic AppliedFiscal year and agency R&D plant Total R&D research research Development R&D plant

Fiscal year 1984 (estimated):Total, all agencies 46,554,924Department of Agriculture 925,364Department of Commerce 367.252Department of Defense 27,987,145Department of Energya 5,770,604Depar tment o f Heal th and Human Serv ices b 4,921,924Depar tment o f the In ter ior 427,558Depar tment o f Transpor ta t ion 538,429Nat ional Aeronaut ics and Space Admin is t ra t ion . , 3,044,400National Science Foundation 1,247,580V e t e r a n s A d m i n i s t r a t i o n 228,100O t h e r a g e n c i e s 1,096,568

Fiscal year 1985 (estimated):Tota l , a l l agencies 54,072,393Department of Agriculture 926,711Department of Commerce 282,357Department of Defense , . 34,510,984D e p a r t m e n t o f E n e r g ya 6,146,700Department of Health and Human Servicesb 4,967,872Department of the Interior 369,209Department of Transportation 505,704National Aeronautics and Space Administration 3,499,400National Science Foundation 1,426,567Veterans Administration 207,600Other aqencies 1,229,289

44,835,777871,942360.021

27,540,0454,825,5764,864,292

421,825515,929

2,888,9001,238,480

220,9001,087,867

52,253,607898,941270,559

34,142,0844,962,2724,953,972

368,989495,204

3,339,4001,414,017

194,5001,213,669

6,981,031386,442

20.522816,590841,671

2,793,052124,667

600689,133

1,172,46615,200

120,688

7,637,587419,727

18,416913,195944517

2,925,916102,762

400826,721

1,335,80915,000

135,124

8,127,270455,594272,644

2,168,1841,231,7331,705,911

276,33081,990

1,012,03166,014

189,700667,139

8,396,633449,981201,187

2,408,2041,268,9641,679,147

248,55679,630

1.088,06378.208

160,000736.693

29,727,47829,90666.855

24,555,2712,752,172

365,32920,828

433,3391,187,738

—16,000

300,040

36,219,38729,23350,956

30,822.6852,748,791

348,90917,671

415,1741 424,616

—19,500

341,852

1,719,14553,422

7,231447,100945,028

57,6325,731

22,500155,500

9,1007,2008,701

1,818, 78627,77011,798

368,9001 184,428

13,900220

10,500160,000

1255013,10015,620

aData shown for flsca[ years 1956.73 and fl.gcal years 1974-76 represent obilgallons of the Atomtc Energy Comm!sslon (A EC) and the Energy Research and DeVelo~mentAdmln!stratlon, respectively

~Data Shown for flscaj years 1955-713 represent obligations of the Depanment of Health, Education, afl~ welfareSOURCE: National Science Foundation

lates to national defense and health, two publicgoods that are not readily measured in economicterms. With the exception of approximately $200million in aeronautics research, all of NASA’s ap-plied research relates to its space activities, whichare not primarily designed to produce economicpayoffs. Only the DOE, Department of “Agricul-ture, Department of the Interior, NASA Aeronau-tics, and Department of Commerce applied re-search programs have primary objectives relatedto improving the economic performance of an in-dustry or the economy as a whole. In sum, nearlytwo-thirds of the Federal applied research budgetis related to the production of public goods, whoseprimary value is not measured in economic terms.

Table 1 reveals another barrier to the use ofeconomic models for research investment in theFederal Government-decentralized planning and

decisionmaking. Six Federal agencies have a shareof the total Federal research budget in excess of5 percent: DHHS, DOD, DOE, NASA, the Na-tional Science Foundation (NSF), and the Depart-ment of Agriculture in descending order. TheOffice of Management and Budget, which coulddevelop an overall national research budget, isdivided into functional directorates that each ex-amine only part of that budget. The Office ofScience and Technology Policy does consider theFederal research budget as a whole, but it has nodecisionmaking authority over Administrationbudget requests.

In Congress, responsibilities are equally dis-persed. Three different authorizing committeesand six largely independent appropriations sub-committees scrutinize the Federal R&D budget ineach House. Thus even if some economic or fi-

6

nancial model could be devised to determine the Government who could ensure its uniform appli-return on the Federal research “investment” and cation across all research fields and budgets. With-serve as a guide to allocating scarce resources, out such a decisionmaker, such a model wouldthere is no single decisionmaker in the Federal have little operational power or efficacy.

BIBLIOMETRICS AND SCIENCE INDICATORSA major problem with the use of economic

models in research decisionmaking is that theydeal with economic “indicators” that are at bestindirectly influenced by research. To measure re-search output more directly, alternative “indica-tors” have been extensively developed by studentsof science policy over the past two decades. How-ever, these have only recently begun to be con-sidered seriously by research policymakers as pos-sible aids to their decision processes. The twomain approaches are bibliometrics, which evalu-ates research output via scientific publications;and science indicators, which measure the vital-ity of the research enterprise in terms of degrees,personnel, awards, and education. Although thesemethods appear to be more appropriate measuresof scientific quality and productivity, they do notoffer the decisionmaker the simple, quantitativeeconomic “bottom line” that economic modelsprovide. The “indicators” can only supplement,and not replace, informed peer judgment of thescientific process. But they can help complete theanecdotal, fragmentary, and, necessarily, some-what self-interested picture of the state of sciencepresented by the researchers themselves. Scienceindicators, and especially bibliometric measures,are reviewed in chapter 3 of this technical memo-randum.

Bibliometrics is based on the assumption thatprogress in science comes from the exchange ofresearch findings, and that the published scien-tific literature is a good indicator of a scientist’sknowledge output. Publications are the mediumof formal information exchange in science and themeans by which scientists stake their claims to in-tellectual “property. ” Therefore, the more pub-lications a scientist has, the greater is his or herpresumed contribution to knowledge.

Simple publication counts have a number of ob-vious flaws: quantity of publications does not

measure the quality of the knowledge containedtherein; publications vary greatly in creativity andimpact. Simple counts also cannot be used forcross-disciplinary analysis because of differencesin publication rates by research field, type of re-search, research institutions, and a number ofother external factors.

Citation analysis addresses the problem ofmeasuring the quality of research output. It as-sumes that the greater the quality, influence, orimportance of a particular publication, the morefrequently it will be cited in the scientific litera-ture. Citation counts based on comprehensivedatabases are being used on a limited basis tomonitor the performance of research programs,facilities and faculties in Europe, and at the Na-tional Institutes of Health (NIH) and NSF.

The problems of citation analysis include: tech-nical problems with the database, variations inthe citation rate over the “life” of a paper, thetreatment of critical or even refutational citations,variations in the citation rate with the type of pa-per, and biases introduced by “self-citation” and“in-house” citations. Developers of the citationsdatabase are working to minimize these problems,but some are inherent. Sophisticated variationson the approach include co-citation and co-wordanalysis, which are described in chapter 3,

Combinations of several research productivityindicators (publications, citation counts, and peerevaluation) have been used in the hope of over-coming problems associated with each method enits own. To the extent that the “partial indicators”converge, proponents argue, the evaluation maybe more meaningful than if only one indicatorwere used. Significant degrees of convergencehave been found by using this methodology toevaluate large physics and astronomy facilities.Since partial indicators depend in large part on

7

a peer review system, they can be used as an in-dependent check on scientists’ peer assessmentsof research activities.

Despite the limitations of bibliometrics, NSFand NIH have undertaken extensive studies to re-fine the techniques and explore their applicabil-ity to research program evaluation. In addition,agencies of the French, Dutch, and British Gov-ernments, and the European Organization forEconomic Cooperation and Development haveapplied some of these indicators to research pro-grams in their countries. The results of thesestudies, and the limitations of this methodology,are discussed in chapter 3.

Science “indicators” assess the ongoing vital-ity of the research enterprise, complementing the“output” measures of bibliometric analyses. Theseindicators include statistics on scientific and engi-neering personnel; graduate students and degreerecipients by field, sector, and institution; and thesupport for graduate education and training. NSF,NIH, and the National Research Council. (NRC)publish detailed indicators on a regular basis.However, the science policy community lacks

consensus on which indicators are most useful orreliable. A report or workshop on the use ofscience indicators to measure the health of the re-search effort in the United States would be a use-ful first step in that direction.

It is important to remember that all measuresor “indicators” of research inevitably are flawed.Any number describing research is an abstractsymbol that depicts, imperfectly, only one aspectof it. Choosing one measure over another impliesthat the measurement user has made some as-sumption about what is important. The chosenmeasure has meaning only through interpretation.

These points underscore the subjective natureof quantitative measures of research—’’objec-tivity” is only apparent. Attaching numbers tosome phenomena allows the expression of certainfeatures in symbols that can be manipulated andconfigured for analysis. This ability is invalua-ble for analytical comparisons and the descrip-tions of trends. Nevertheless, a number remainsno more than an abstract symbol that someonedecided best captures a particular aspect of somereal-world phenomena.

RESEARCH DECISIONMAKING IN INDUSTRY AND GOVERNMENT

To determine the degree to which economicand noneconomic techniques are used by researchmanagers today, OTA reviewed the literature onthe use of these techniques in industry and gov-ernment and interviewed experienced officials inboth sectors. A list of those techniques is providedin table 2. The findings were quite surprising.

In industry, where one might expect quantita-tive techniques to prevail due to the existence ofa well-defined economic objective for the individ-ual firm or business, OTA found great skepticismamong research managers about the utility of suchtechniques. Managers found them to be overlysimplistic, inaccurate, misleading, and subject toserious misinterpretation. At the project selectionand program evaluation levels, there is little sys-tematic data about the use of quantitative tech-niques. Most articles describe a process adoptedby one firm or another without any indication as

to how widespread the practice is in industry asa whole. This literature is reviewed in chapter 4.

Peer review dominates program evaluation inindustry, with an occasional firm attempting bib-liometric analyses. For project selection, firms usestandard economic return on investment tech-niques for projects at the development end of thecycle, where costs and benefits are generally wellknown and the risk can be quantified using pastexperience. At the basic research end of the spec-trum, industry’s project selection techniques tendto be quite subjective and informal, supplementedoccasionally by scoring models. (See ch. 4 for defi-nitions of the different techniques used in projectselection. ) At the applied research or exploratorydevelopment stage, simple, unsophisticated selec-tion procedures, based on a page or two of qual-itative information or a simple rating scheme,dominate.

Table 2.—Quantitative Methods UsedTo Evaluate R&D Funding

Economlc (measure output in terms of dollars or productivity). Macroeconomic production function (macroeconomic)● Investment analysis

–Return on investment (ROI)–Cost/benefit analysis (CBA)—Rate of return—Business opportunity

● Consumer and producer surplus

Output (measure output in terms of published information)● Bibliometric (publication count, citation, and co-citation

analysis)● Patent count and analysis• Converging partial indicators• Science indicators

Project selection models. Scoring models● Economic models● Portfolio analysis (constrained optimization). Risk analysis and decision analysis

SOURCE: Office of Technology Assessment.

At the level of strategic planning and resourceallocation for R&D in industry, some interestingpatterns have emerged. Industry tended to fundresearch somewhat unquestioningly in the 1950sand 1960s, only to become skeptical of a lack ofdemonstrable return on the investment in the1970s. Each industry tended to have a rule ofthumb; R&D should bes percent of sales, or per-haps 10 percent in an R&D-intensive industry. Inthe 1970s, corporate strategic planning came intovogue, and technological change came to be rec-ognized as an integral part of corporate planning.R&D planning and budgeting was integrated intothe overall corporate strategic effort. Many firmsset up committees and other formal mechanismsto assess long-term technical opportunities, estab-lish broad goals for the commitment of resources,ensure that resources are properly allocated to de-velop the technology necessary to support thosegoals, approve major new product programs, andmonitor progress.

The primary goal of such committees appearsto have been to ensure that R&D managers com-municate regularly and formally with planning,financing, marketing, manufacturing, and otherconcerned parts of the corporation in setting andachieving technological goals. Corporate manag-ers have learned that R&D planning and budget-

ing is primarily an information and communica-tion process, involving many persons and manylevels of the corporate hierarchy, using many cri-teria and several iterations. The goal of corporatemanagers has been to improve communication byinvolving all affected parties. Economic and fi-nancial modeling appear to play a secondary rolein this process, seining primarily as inputs to over-all corporate strategic planning.

Government R&D managers also avoid quanti-tative techniques for project selection and pro-gram evaluation. Surveys of government researchmanagers reveal little use of quantitative meth-ods for choosing projects or evaluating program:(some notable exceptions are discussed inch. S)Peer review tends to be the preferred methodof project selection, with the term “peer” oftenbroadened to include agency technical staff. Bib-liometric techniques have been used extensive>by NIH and on a limited basis by NSF in programevaluation. NASA and the National Bureau ofStandards have carried out economic return-oninvestment analyses, with limited success.

Budgeting for research and development sharemany of the characteristics of traditional Federabudgeting. It is incremental, fragmented, specialized, repetitive, and based, to a large degree, orrecent history and experience. Much of the budgeting is carried out by experts in narrow specialties, who focus their attention on increments toexisting base programs. Attempts to “rationalizethe system by introducing techniques such a“program planning and budgeting (PPB)” and“zero based budgeting (ZBB)” have largely bee:abandoned as unworkable and inappropriategiven the political nature of the Federal budgeprocess.

Some R&D forecasting and strategic planningis carried out by agency advisory committeessuch as NSF’s National Science Board, DOEEnergy Research Advisory Board, and NHI’s advi-sory councils. NRC and its constituent bodies for-really review Federal research programs and pro-duce Research Briefings and Five-Year Outlookthat identify promising new avenues of researchNone of those efforts constitutes true strategicplanning or forecasting. The Japanese, however

9

provide an interesting model of systematic fore-casting and planning for R&D.

Chapter 5 also describes the R&D forecastingcarried out by Japan’s Science and TechnologyAgency and Ministry of International Trade andIndustry. These forecasts identify research areasof long-term strategic importance using back-ground information on research trends gleanedfrom industry, government, and academic reportsfrom around the world. They incorporate “tech-nology-push” and “market-pull” perspectives byinvolving both laboratory researchers and indus-trial users, and utilize a bottom-up rather thana topdown approach, drawing heavily on recom-mendations from the affected communities. Theprocess provides a forum for people from differentgroups and different professions to communicateabout R&D priorities. This enables policymakers,professional forecasters, scientific analysts, andacademic and industrial researchers to coordinateresearch plans and to form a consensus on pri-orities for future strategic research. Participantshave a stake in the successful outcome and follow-through of the process, which tends to make theforecasts self-fulfilling. The emphasis on commu-nication and involvement of all affected partiesis strikingly similar to the lessons learned by U.S.corporate management with respect to R&D plan-ning and resource allocation described above.

SUMMARY

In summary, OTA finds that the metaphor ofresearch funding as an investment, while validconceptually, does not provide a useful practicalguide to improving Federal research decisionmak-ing. The factors that need to be taken into accountin research planning, budgeting, resource alloca-tion, and evaluation are too complex and subjec-tive; the payoffs too diverse and incommensur-able; and the institutional barriers too formidableto allow quantitative models to take the place ofmature, informed judgment. Bibliometric andother science indicators can be of some assistance,

The review of industry and government R&Ddecisionmaking presented in chapters 4 and 5leads to two conclusions. First, R&D managementand resource allocation are complex decision-making processes involving trade-offs betweenfactors that often cannot be precisely measuredor quantified. Any effort to substitute formalisticquantitative models for the judgment of mature,experienced managers can reduce rather than im-prove the quality of R&D decisionmaking. Theresistance of R&D managers to the use of quan-titative decision tools is, to some degree, a rationalresponse to the complexity and uncertainty of theprocess.

Second, the process of decisionmaking canoften be as important as the outcome. In both theU.S. and Japanese cases, bringing together expertsfrom a variety of fields and sectors and provid-ing them with a vehicle to discuss R&O priori-ties, budgets, and plans, was critical to success.It may be that discussions of R&D resource allo-cations in the United States should focus less onthe overall numbers and more on the process bywhich those numbers were generated, with spe-cial attention paid to questions of stakeholder in-volvement and communications.

especially in research program evaluation, andshould be used more widely. However, they areextremely limited in their applicability to inter-field comparisons and future planning. The re-search planning and budgeting experience in someU.S. corporations and the R&D forecasting effortsin Japan suggest a need to improve communica-tion between the parties that carry out and uti-lize research, and to assure that a wide range ofstakeholders, points of view, and sources of in-formation are taken into account in formulatingR&D plans and budgets.

Chapter 2

Measuring the Economic Return:Progress and Problems

Chapter 2

Measuring the Economic Returns:Progress and Problems

INTRODUCTION

This chapter discusses past and current use ofeconomic measures of the return on research anddevelopment (R&D) as an investment in individ-ual industries and at the national level. Differentapproaches distinguish between direct and indirectreturns, private and Federal spending, basic andapplied R&D. This chapter examines the difficul-ties and ambiguities encountered when trying toextend the analysis of private sector R&D spend-ing to Federal R&D investments. It reviews at-tempts— of mixed success-to use econometricmethods to measure the returns on Federal R&Ddollars in three industries that have been well-studied: agriculture, aviation, and health.

Federal R&D dollars may also have indirect ef-fects on productivity by triggering spinoffs orspillovers; the chapter looks at the National Aer-onautics and Space Administration (NASA) as anexample of the use of econometric methods tomeasure such indirect effects. The chapter con-cludes that while econometric methods have beenuseful to track private R&D investment within in-dustries, the methods fail to produce consistentand useful results when applied to Federal R&Dsupport. From these findings, OTA concludes thateconomic investment models are not likely to beof great utility in helping to guide Federal researchdecisionmaking .

ECONOMETRIC STUDIES AT THE NATIONAL LEVEL

Over the past three decades economists have E.F. Denisen attributed 20 percent of the growthattempted to investigate the effect of research in the Nation’s real income between 1929 and 1957expenditures, technological change, and other to “advance of knowledge” and 11 percent toresearch-related inputs to production on the economies of scale.1 Such studies indicated thatgrowth of GNP, productivity, and employment. technological change, however defined, is impor-Their basic production function approach has tant to national economic performance. z

been to separate the inputs to the economy intothree groups: capital and labor supply—the ma-jor factors determining the productivity of a firm,industry, or national economy—and an “otherfactors” category, assumed to account for allchanges in productivity that could not be ex-plained by changes in labor and capital. Thisresiduals category includes scientific knowledge,technological advance, managerial and market-ing expertise, economies of scale, the health andeducation of the work force, and other factors that

Private R&D Investment

Building 0,1 these studies, in the late 19 SOS

economists began to include R&D expenditures(assumed to be a rough indicator of technologi-cal advance) as an input to their productivity cal-culations, along with capital and labor. Numer-ous studies found a strong correlation betweenR&D spending and productivity growth. Look-ing at R&D as an investment, economists sought

affect the efficiency of resource use.‘E.F. Denisen, The SOIMCS of Economic Growth ]n the Lf.S

In the 1950s, economists recognized that resid- (New York: National Bureau ot Economic Research, 1Q62).2ZVI Griliches, “Issues m t%sessmg the Contribution ot Researchual factors were a major influence in economic and Development to Productlwty Growth l%e Beii Iournai clt EC-LP

growth. Using the “factor productivity method, ” nomlcs, vol. 10, spring 197Q.

13

14

to measure its rate of return. Fellner3 calculateda 31 to 55 percent rate of return for the entireeconomy. Terleckyj4 estimated a 29 percent re-turn to firm-financed R&D. Mansfield5 estimateda 40 to 60 percent return in the chemical industryand Link* estimated 21 percent in the petroleumindustry. More recent in-depth studies confirm thecorrelation between private R&D spending andproductivity increases (see box A).

These studies are representative of the strongand consistently positive correlation found be-tween privately financed R&D and productivitygrowth in the manufacturing industries. They sug-gest that econometric analysis of private R&Dspending produces estimates useful in evaluationand planning. However, the wide range of cal-culated rates of return to R&D spending and theinability to assign causality to the correlations re-flect the tentative and hypothetical nature of themethodologies. Each study works with differentassumptions and definitions. Results are most de-finitive and consistent for private spending withinone firm or industry, where it is easiest to defineand measure inputs and outputs.

Social return often exceeds the private rate ofreturn, as a company doing the R&D cannot reapall the benefits from its work. One industry’s R&Dcan spin off substantial benefits to other indus-tries and other sectors of society, a difficult out-put to quantify. In studies by Mansfield andothers, the social rate of return was two or moretimes the private rate of return.7

In examining the applications of these economicmodels it should be kept in mind that they areonly hypothetical constructs that attempt to de-scribe complex events. Zvi Griliches, one of the

‘W. Fellner ‘Trends in the Activities Generating TechnologicalProgress, ” American Economic Review, vol. 60, March 1970, pp.1-29.

‘Nestor E. Terleckyj, i%xts of R&Don the Productivity Growth ofMustries: h &xplorato~ Study (Washington, DC: National Plan-ning Association, 1974).

‘Edwin Mansfield, “Rates of Return From Industrial Researchand Development, ” American Economic Rew”ew, vol. 55, May 196.S,pp. 310-322.

‘A.N. Link, “Productivity Growth, Environmental Regulationsand the Compmition of R& D,” The Belf Journal of Economics, vol.13, autumn 1%2, pp. 166-l@.

“Edwin Mansfield, “The Economics of lnnovatlon, ” Innovationand U.S. Research, W. Novis Smith and Charles F. Larson (eds. ),ACS Symposium Series 129, Washington, DC, 1980, pp. 96-97.

foremost users of these models, warns that theequations in the models reveal correlation, notcausality.8 Nor do the models reveal the path bywhich R&D investment allegedly leads to produc-tivity improvements. Moreover, the need to treatR&D as the “residual,” or “the thing that remainsafter everything else is accounted for, ” furtherweakens the proof of relationship, since it is en-tirely possible that other components of the re-sidual exist, but have not been included in theanalysis. Finally, the production function ap-proach of neoclassical economics is simply anhypothesis about the way the world works; it hasnot been proven that such production functionsexist or take the form assumed by economists. Forall these reasons the impressive returns on privatesector R&D investment reported above should beviewed with caution.

The Returns, or Lack Thereof,to Federally Funded R&Din Specific Industries

Econometric approaches have been unsuccessfulin establishing a return on federally funded R&D.Unlike the strong and consistently positive corre-lations found between privately financed R&Dand productivity growth in the manufacturing in-dustries, only weak and inconsistent correlationshave been found for federally funded R&D. Ter-leckyj, in the 1975 study reported above, foundthat for the 20 manufacturing industries he stud-ied, “the coefficients for government-financedR&D are not statistically significant, and the co-efficient for government-financed R&D performedin industry is actually negative. ”9 A decade laterTerleckyj reported subsequent studies that con-firmed the weak indicators and smaller effects ofgovernment-funded R&D. 10 Even in two indus-trial sectors enjoying high, long-term governmentfunding and interest—aircraft manufacturing, andcommunication and electronic components—Ter-

‘Ibid., p. 24, emphasis added.Wester E. Terleckyj (cd.), State of Science and Research: Some

New Indicators (Boulder, CO: Westview Press, 1977), p. 131.“’Nester E. Terleckyj, “Measur]ng Economic Effects of Federal

R&D Expenditures: Recent History With Special Emphasis on Fed-eral R&D Performed in Industry, ” paper presented to the Nat]onalAcademy of Sciences Workshop on “The Federal Role ]n Researchand Development, ” Nov. 21-22, 1985, p. 5.

.

15

Box A.—Recent Research Productivity Studies

Nestor Terleckyj, studying the productivity of entire industries in 1974, found that an industry’s rateof productivity increase is directly related to the amount of its own R&D and to the amount of R&D car-ried out by its supplier industries.1 In a study of the relationship between total factor productivity andR&D in 33 manufacturing and nonmanufacturing industries between 1948 and 1966, Terleckyj estimateda 28 percent productivity return on private l&D investment in the manufacture“ g industries. He foundan even higher implicit productivity return on company-sponsored R&D by taking into account the R&Dinherent in purchases from supplier industries. For the nonmanufacturing industries the correlation wasmuch weaker, and in some cases actually negative. z

Zvi Griliches, in a 1975 study of 883 companies representing more than 80 percent of all the industrialR&D conducted in the United States, found a .7 percent rate of return to total R&D, private plus gover-nment funded, for the period 1957-65. There was a wide range in the rate of return by industry, with thechemical industry at the top at 93 percent; electric equipment and aircraft and missiles at the bottom at3 to 5 percent; and metals, machinery, and motor vehicles in the middle at 23 to 25 percent. For privatelyfinanced R&D alone, Griliches found a substantially higher average return of 32 to 40 percent.3 Terleckyjfound this return to be quite comparable to his own value for the manufacturing industries of 37 percentreturn on private R&D when only direct R&D inputs were considered.’

Griliches, in a follow-up to his 1975 study, found that firms that spend a larger fraction of their R&Don basic research are more productive.5 He found that basic research had 2.5 to 4.5 times as great an effecton productivity per dollar invested as total R&D. However, he cautioned that R&D or basic research maynot drive productivity and profitability successes, but the correlation could well be that “success allowsfirms to indulge in these types of luxury pursuits.”

Edwin Mansfield, the third major analyst in this field, refined Terleckyj’s work on the 20 manufactur-ing industries by dividing R&D into its basic and applied components. He found a “strong relationshipbetween the amount of basic research carried out by an industry and the industry’s rate of productivityincrease during 1948-1966.”6 In a further study of 37 innovations Mansfield compared the return on R&Dfor those innovations to the firm making the investment (the “private return”) with the return to societyas a whole (the “social return”). He found a median private rate of return of about 25 percent, but a mediansocial return of close to 70 percent.7

Iikhvin Mansfield, “Research and *iopment, Productivlt’; , and Inflation, ” Science, vol. 209, Sept. 5, 1960, p. 1,091.?+stor E. Terkckyj (cd.), “Estimates of the Direct and Inmrect Effects of 1ndus~al R&D on Economic Growth” ~e state Of s~en~ and RE-

+T Some NW hd.katofi (Boulder, CO: WestView press, 1‘~, PP. l~lqz.‘Ibid., pp. 133134.‘Ibid., p. W.‘Zvi Griliches, “Productivity, R&D, and Basic Research at tb- Firm Level in the 1970s, ” NBER Working Paper No. 1S47, typescript (National Bu-

reau of Econom;c Affairs, 10s0 Massachusetts Avenue, Cambn{lm, MA 021W), January 19~ P. 16.‘%dwm Mansfield, “Basic Research and Productivity Iricreaw m Manufactunn g,” American Economrc Rewew, vol. 60, No 5, December 1960.

p. 866‘Edwin Mansfield, “How Economists See R& D,” Reseamh management, July 1982, p. 27.

leckyj “found strong positive association betweenprivate R&D and productivity,” but “no effect ofgovernment R&D.” 11

Measuring Federal R&D spending is more com-plex than in the private sector. Tracing outputsthrough the long and nebulous path from basicresearch to commercial product is especially dif-ficult. A company does research aimed at a pe-

cific product or market, controls the entire prod-uct development process, manages its marketing,and has a clear record of inputs and outputs. Fed-eral research managers do not target R&D sosharply, have virtually no say in private sectordecisions to develop a product, and have no in-fluence and often no knowledge of what is hap-pening in the market.

Terleckyj attributes the failure to find a returnon federally sponsored industrial R&D to the fact

16

that “government funded industrial R&D is a pub-lic good and therefore is used by all users to theextent where its marginal product is zero.” There-fore, according to Terleckyj, “its contribution toproductivity cannot be observed statistically bytraditional techniques and approaches. ”12 In ad-dition:

there is an inherent conceptual limitation inthe national income accounting (and the GNPdata) in that it attempts to measure the real costand the real product of the public sector at thesame time. While the resource cost utilized in thepublic sector can be identified, the real output ofpublic goods cannot be measured because theirmarginal product and implicit price is alwayszero. 13

The inability to find a meaningful correlationbetween government-funded R&D and productiv-ity increases in the economy as a whole has ledeconomists to examine more closely the indirectimpact of Federal R&D on privately funded in-dustrial R&D. According to Terleckyj, studiesdone in the past 6 years “indicate that in mostcases government R&D expenditures have beenpositively related to private R&D expenditures. ”14

Peter Reiss, reviewing the same literature, reports12Ibid., p. 7.13Ibid., p. 8.14Ibid., p. 9.

a “general impression that Federal R&D is a com-plement to private R&D efforts, ” but finds a lackof “very good conceptual models of how FederalR&D affects private R&D incentives. ”15

Frank Lichtenberg has attempted to distinguishthe direct and indirect links between Federal aridprivate R&D. He argues that Federal R&D ex-penditures “may, in principle, increase the aver-age and marginal cost of private R&D performa-nce by driving up the prices of R&D inputs’’—or“crowding out” private R&D. Alternatively, fed-erally sponsored R&D may leverage private R&D,reducing the costs of private research and inno-vation and raising the productivity of privateR&D—the “spillover effect.” He finds econometricevidence for the crowding out hypothesis in theshort run, although less so in the long run. Hefinds limited evidence for cost-reducing spilloversbut concludes that “it is probably the case thata small fraction of federally supported R&D gen-erates very large spillovers (some of which maybe negative.)’’”

“Peter C. Reiss, “Economic Measures of the Returns to- FederalR& D,” paper presented at the National Academy of Sciences Work-shop on ‘The Federal Role in Research and Development, ” Nov.21-22, 1985, pp. 11-12.

‘*Frank R. Lichtenberg, “Assessing the Impact of Federal indus-trial R&D Expenditures on Private R&D Activity, ” paper pr=ntedto the National Academy of Sciences Workshop on “The FederalRole in Research and Development, ” Nov. 21-22, 1985, pp. 31-32.

INDUSTRY CASE STUDIES: AGRICULTURE, AVIATION, AND HEALTH

Despite the problems in linking governmentR&D expenditures to productivity improvementsin the economy as a whole, studies have shownsector-specific productivity improvements fromtargeted government R&D programs. This sectionlooks at econometric analyses of Federal R&Dsupport of three industries-agriculture, aviation,and health—whose long and heavy dependenceon Federal R&D financing has made it feasible foreconomists to estimate inputs, outputs, and ratesof return. The results of, and problems with, thoseevaluations are presented below.

Agriculture

For nearly a century, since the passage of theHatch Agricultural Experiment Station Act in1887, the Federal Government has had a programto support applied research related to improvedfarm productivity. The program today has threemain elements: 1) the Agricultural Research Serv-ice, which funds research and technology trans-fer projects at the USDA’s own research stationsand at state universities; 2) the Cooperative Re-search Service, which consists primarily of match-

17

ing (Hatch Act) grants to State Agricultural Ex-periment Stations located at the State Land GrantUniversities set up by the Land Grant College Actof 1864; and 3) the Economic Research Service.The budgets for the three services for fiscal year1985 were $470 million, $320 million, and $50 mil-lion, respectively.

17 In addition, the States pro-

vide more than $500 million a year in researchfunding to their agricultural experiment stations.The evolution of Federal and State support forresearch and extension directed at improvementsin agricultural production technology is presentedin table 3 below.l8

Many econometric studies of the productivityreturn to agricultural research have been carriedout in the past three decades, beginning with Zvi

“National Science Foundation, Federal R&D Fund”ng by BudgetFunction, Fiscal Years 1984-1986, NSF 85-318 (Washington, DC:NSF, March 1985), pp. 74-75.

‘SR.E. Evenson, “Agncuhure,” ch. s of Richard R. Nelson ed. ),Government and Technical Progress: A Cross-Industry An.dvsIs(New York: Pergamon Press, 1982), p. 252.

Table 3.— Expenditures by the Public Sector onResearch and Extension Oriented to ImprovedAgricultural Production Technology, 1890.1970

Expenditures on researchState agricultural experiment stations

USDA Expendituresfunded on Public

% State % Federally % USDA outside ExtensionYear Total a funded funded funded Statea Service a

1890 97 22 78 2.61900 12.2 34 66 10.41910 370 39 61 4 7 41915 342 72 28 62.51920 287 77 23 49,01925 425 85 15 59.21930 75.6 73 27 96.51935 793 57 27 16 6 6 51940 113,2 54 28 18 1199? 945 1142 56 23 20 9 7 8? 950 1943 63 17 20 8 3 41955 251 4 63 17 20 89.21960 3448 55 15 30 8 7 61965 385.5 58 16 26 6 7 81970 414,5 66 16 18 109.51975. 420.0 na na na 110,01980. 428.0 na na na 110.0

0.31 32 4

18,04 6 461 67 7 27 0 2

107. 71020140.8152 1169.5179.72210264.0314.0

aln mllllon~ of constant 1980 dollars

SOURCE R E Evenson, “Agriculture,” ch. 5 of Richard R Nelson (cd.), Govern.rnent and Techmcal Progress” A cross-lndusf~ Analysts (New YorkPergamon Press, 1982), p 252

Griliches’ classic 1958 study of hybrid corn tech-nology. All but one of the studies have shown avery high internal rate of return on public sectoragricultural research, as can be seen from table4 below. The rate of return varies from a low of21 percent to a high of 110 percent, with the vastmajority in the 33 to 66 percent range. Public sec-tor agricultural research has generally been con-sidered to have been a significant success. RichardNelson summarizes the characteristics of the agri-

Table 4.—Econometric Studies of Productivity Returnto Agricultural Research in the United States

Author (date) Commodity Time period Rate of return

Gri l iches (1964) Aggregate 1949-59 35-40output

L a t i m e r ( 1 9 6 4 ) Aggregate 1949-59 Not significantoutput

Peterson (1967) Poultry 1915-60 21Evenson (1968) Aggregate 1949-59 47Cline (1975) Aggregate 1939-48 41-50Knutson and Tweeten

( 1 9 7 9 ) , , Aggregate 1949-58 39-471959-68 32-391969-72 28-35

Bredahl and Peterson(1976) Cash grain 1969 36

Poultry 1969 37Dairy 1969 43Livestock 1969 47

Davis (1979) Aggregate 1949-59 66-1001964-74 37

Evenson (1979) Aggregate 1868-1926 651927-50 95 (applied R&D)1927-50 110 (basic R&D)1948-71 45 (basic R&D)

Davis and Peterson(1981) Aggregate 1949 100

1954 791959 661964 371969 371974 37

Norton (1981 ) Cash grain 1969 3 1a

Poultry 1969 27Dairy 1969 56Livestock 1969 30Cash grain 1974 44Poultry 1974 33Dairy 1974 66

aBa9@ on maximum lag length estimated (9 Yearsl

SOURCE: Robert D Weaver, “Federal R&D and U S Agriculture An Assessmentof the Role and Productwty Effects, ‘ paper presented at the NationalAcademy of Sciences Workshop on “The Federal Role In Research andDevelopment, ” Nov 21-22 1985, p 27

18

cultural sector that have made it amenable to thissuccess:

In the first place, farming is an atomistic indus-try and farmers are not in competition with eachother. Differential access to certain kinds of tech-nological knowledge, or property rights in certaintechnologies, are not important to individualfarmers. This fact at once means that farmers havelittle incentive to engage in R&D on their own be-half and opens the possibility that the farmingcommunity itself would provide the political con-stituency for public support of R&D.

The Federal/State agricultural extension system. . . marshaled that support and put the farmersin a position of evaluating and influencing thepublicly funded applied R&D. The system ishighly decentralized. The regional nature of agri-cultural technology means that farmers in indi-vidual states see it to their advantage that theirparticular technologies be advanced as rapidly aspossible.

It was [the] combination of an evolving set ofagricultural sciences based in the universities andsupported publicly, and applied research and de-velopment also publicly funded but monitoredpolitically by the farming community, that hasmade public support of agricultural technologyas successful as it has been. Where private com-panies are funding significant amounts of inno-vative work and the industry is reasonably com-petitive, it is in the interest of the fanners as wellas the companies that public R&D money be al-located to other things. [A] reasonably well de-fined division of labor has emerged between pub-licly and privately funded applied research.l9

The nature of the agricultural sector explains whyFederal R&D has a powerful effect and why econ-ometric methods can arrive at relatively reliableestimates of this effect.

Aviation

Since World War II the Federal Governmenthas provided a considerable amount of R&D sup-port for aviation. Indeed, according to DavidMowery, “the commercial aircraft industry is vir-tually unique among manufacturing industries inthat a Federal research organization, the National

“Richard R. Nelson (cd.), Government and Technical Progress:A Cross-fndustry haiysis (New York: PergarnOn F’mst 19821, PP.466-467.

Advisory Committee on Aeronautics (NACA,subsequently the National Aeronautics and SpaceAdministration, NASA) has for many years con-ducted and funded research on airframe andpropulsion technologies.”20 In addition, the De-partment of Defense has provided considerablesupport for research and development on militaryaviation that has generated considerable civilianspinoffs, and the aircraft industry itself conductsa great deal of in-house R&D. The total nationalR&D expenditure for aircraft from 1945 to 1982was $104 billion (in 1972 dollars), of which $77billion was provided by the military, $9 billionby NACA/NASA and $17.4 million by industry.Figure 1 breaks out those expenditures by year.About 45 percent of the total R&D budget wentto aircrames, about 30 percent to avionics, andabout 25 percent to engines.

The benefits of this investment to the U.S. avia-tion industry and the consuming public were sub-stantial. According to Mowery, the “total factorproductivity in this [the commercial aviation] in-dustry has grown more rapidly than in virtuallyany other U.S. industry during the postwar pe-riod. ”2l Two commonly used indices of aircraftperformance are the number of available seatsmultiplied by the cruising speed (AS X Vc) andthe direct operating costs per available seat mile(DOC). Between the DC-3 of 1940 and the Boe-ing 707 of 1959 the AS X Vc increased by a factorof 20 and the DOC fell by a factor of 3. The in-troduction of the Boeing 747 in 1970 increased theAS X Vc by another factor of 3 and halved directoperating costs. According to Mowery’s calcula-tions, if the total volume of airline passenger traf-fic in 1983 were to have been flown using 1939technology (primarily the DC-3), the cost to theNation would have been $24 billion (in 1972 dol-lars) rather than the $5.8 billion actually incurred(also in 1972 dollars). Thus, improvements incommercial aircraft technology led to more than$18.2 billion (in 1972 dollars) in additional airtransportation services rendered for the actualamount paid. This benefit is considerably over-stated in that consumers would have undoubtedly

~avid–C. Mowery, “Federal Funding of R&D In Transporta-tion: The Case of Aviation, ” paper presented to the National Academy of Sclenc= Workshop on ‘The Federal Role In Research an~Development, ” Nov. 21-22, 1985, p. 13.

“Ibid., p. 6.

19

Figure 1.— Annual R&D Investment, 1945-82(1972 dollars)

3,500 .

.

.

MilitaryR&D

1945 1950 1955 1960 1965 1970 1975 1980Year

SOURCE: Dawd C. Mowew, “Federal Funding of R6D in Transportation: The Caseof Avtatlon, ” paper presented to the National Academy of SciencesWorkshop on “The Federal Role In Research and Development, ” Nov.21-22, 1985.

used other modes of transportation, or foregonea considerable amount of travel, had the aircraftoperating costs not declined so substantially. Onthe other hand, the benefit does not take into ac-count the value of the time saved by more rapidairline travel, the additional economic activitygenerated by an expanded airline industry, andthe foreign trade benefits of the multibillion dollarsales of U.S. aircraft abroad.

If the $18 billion in additional air transporta-tion services is taken as the benefit of improvedaircraft technology, and the $104 billion total in-dustry plus government R&D expenditure from1945 to 1982 as the cost, then the social returnfrom this investment appears to be on the order

of 17 percent per year. If military developmentexpenditures—about half of total military avia-tion R&D costs—am subtracted from the costside, on the grounds that they do not directly sup-port the commercial market, then the return oninvestment increases to close to 27 percent peryear. Mowery has carried out a more sophisti-cated return on investment calculation in whichhe finds the internal rate of return from industry-financed and civilian Federal R&D to be about24 percent.

Mowery also emphasizes that other factors playan important role. Civil Aeronautics Board reg-ulations encouraged the adoption of new aircrafttechnologies between 1938 and 1978 by control-ling air fares, which encouraged the airline indus-try to pursue a marketing strategy of technicalinnovation and service improvements. The reg-ulatory incentive to innovate probably amplifiedthe apparent social return.

Although it is not possible to isolate the civil-ian return on Federal aviation R&D, the dramaticexpansion of the airline and aircraft industries inthe United States after World War II is clear in-dication of the benefits of this unique Federal sec-toral policy. Even in this industry, the economistsare not unanimous on the productivity benefits.TerIeckyj, for example, claims to have found “noeffect of government R&D” on productivity im-provements in the airline industry.22 That hisconclusions clash with common sense and otheranalyses illustrates the danger of depending solelyon economic formulas to guide Federal R&D de-cisions.

Health

In 1985, the United States spent $381.2 billionon health care. Three percent of that, $11.5 bil-lion, went to health R&D. The Federal Govern-ment funded two-thirds of the research, with halfthe money going to the National Institutes ofHealth (NIH).23 With health care accounting formore than 10 percent of GNP and health research

‘Terleckyj, “Measuring Ecorlomic Effects of Federal R&D Ex-penditures: Recent History With Special Emphasis on Federal R & DPeriormed In Industv, ” op. cit., p. ~

‘U.S. Department of Health and Human Services, .N’IH DataBook, MH Publication No. 85-1261 (Washington, DC: DHHS, June1985).

2 0

claiming 33 percent of all nondefense Federal R&Dfunds,24 health policymakers have asked whetherit is possible to measure the economic benefits ofbiomedical research and development in terms ofthe primary output, improvements in health.

Measuring the productivity of health R&D iscomplicated by the value-laden issue of settingcomparable economic values on well-being, ill-nesses, diseases, and life span. The economic costsof illness and disease, embodied in a “cost of ill-ness” model, inform the national health agendaand the allocation of Federal health researchbudgets. An agency of the Public Health Service,the National Center for Health Statistics, has beendirected by Congress to calculate annually theeconomic costs of illness and disease.

In response to an NIH initiative in the early1970s, Selma Mushkin and her colleagues at-tempted to use a human capital methodology toquantify the economic costs of disease between1900 and 1975.25 They first calculated straightfor-ward direct costs: expenditures on hospital care,physician’s services, nursing home care, drugs,and medical appliances. The human capital modelalso includes indirect costs, specifically morbid-ity costs, which are losses incurred when illnessresults in absence from employment or a disabil-ity removes someone from the work force; andmortality costs, losses due to premature death.Mortality costs, in particular, embody the humancapital approach in that they value one’s lifeaccording to one’s earnings, or according to themarket value of one’s skills. They estimate presentvalue of future losses due to premature death, tak-ing into account life expectancies for different sex,race, and age groups, varying labor force partici-pation rates, and the discount rate.26

Mushkin attempted to estimate the contribu-tion to the observed reduction in the national mor-

“American Association for the Advancement of Science, Re-search & Development, FY 1986 (Washington, DC: AAAS, 1985),p. 27.

“Selma Mushkin, Biomedical Research: Costs and Benefits(Cambridge, MA: Ballinger Publishing Co., 1979).

‘bFor extensive illustrations of the human<apltal approach, see,B.A. Weisbrod, “Costs and Benefits of Medical Research: A CaseStudy of Poliomyelitis, ” Jouma/ of Political Economy, vol. 7Q, hlay -June 1971, pp. 527-544. .AIso R. FeIn, Economics ok Mentaf Illness(New York: Basic Books, 1958). Also, S.J. hlushkin, “Health as anInvestment, ” ]oumaf of Po/itjcaf Economy, vol. 70, October 1962,pp. 129-159.

tality rate of a number of factors, such as betterpublic health measures, improved working con-ditions and nutrition, better personal health careand higher income, and biomedical research. Shetreated the mortality rate as a function of five in-dependent variables—economic factors, societalfactors, environmental factors, provider charac-teristics, and a measure of technical advances at-tributable to biomedical research and develop-ment. She then used regression analyses and otherstatistical techniques to determine the coefficientsof each of the five variables in the equation pre-dicting mortality rates. Indicators of technical ad-vances due to research and development provedespecially difficult to find, so Mushkin and hercolleagues were forced to treat the technologyvariable as a residual; any reduction in mortal-ity not attributable to the other variables in thefunction was attributed to biomedical research.

Based on this model, Mushkin found that bio-medical research accounted for 20 to 30 percentof the reduction in mortality between 1930 and1975. She estimated that each l-percent increasein biomedical research funds lowered mortalityrates 0.05 percent. She also estimated that 39 per-cent of the reduction in days lost due to illnesscould be attributed to the results of biomedicalresearch. Using human capital theory she esti-mated the value of each premature death avertedat $76,000 and each work-year gained when ill-ness is averted at $12,250. With these dollar valuesinserted for the reduced mortality and morbidityattributed to biomedical research, Mushkin founda return of $145 to $167 billion on a $30 billioninvestment, equivalent to an internal rate of re-turn of 46 percent.

Critics have attacked the cost of illness modelon economic and ethical grounds. Stephen Strick-land argues that applying such estimates to deci-sions on public spending “carries an unacceptableimplication that people should be protected, orsaved, in proportion to their economic produc-tivity and personal earnings. ”27 Such an approachdevalues or dismisses the lives of many older citi-zens, children, women working in the home, orunderemployed minority groups. In addition, aperson’s earnings may vary significantly over

‘“Stephen P Strickland, Research and the Health of .4merlcans(Lexington, MA: Lenngton Books, 1!?78), p. 4S

21

time, making extrapolations unrealistic. Further-more, depending on the data and methodologyused, calculations of the cost of a disease to theeconomy can produce a wide range of conclu-sions. Table 5 illustrates the divergent estimatesof three groups analyzing the costs of the samediseases.

Many important costs of illness are also lessquantifiable, such as the psychological costs forthe patient and family. Costs can extend beyondthe immediate period of illness and go unac-counted for if the disease is chronic. Some illnessesentail intergenerational costs, creating immeasura-ble long-term effects.

Measuring productivity gains in health careraises additional questions for economists. First,how does one assign a value to the health of in-dividuals? Because almost all medical bills are paidby a third party-the government or an insurancecompany—health care is not as subject to mar-ket forces as other industries, and it is impossi-

ble to know the value of the care to the consumer.One indirect measure of the economic benefits ofpublic health is increased worker productivity, butthis gives no value to the health of the retired orthose outside the work force. Distinguishing theinputs to improved health is as difficult as meas-uring the outputs; changes may be due to im-proved biomedical technology, nutrition, environ-mental conditions, exercise habits, more widely

available or affordable health care, or a host ofother factors. 28

Physician and economist Jeffrey R. Harris sum-marizes the knotty conceptual and methodologi-cal problems inherent in measuring economicreturns on investment in biomedical research:

1.

2.

3.

4.

5.

6.

In the biomedical sciences, the separate con-tributions of basic and applied research in bio-medicine can be difficult to distinguish. . . .The separate contributions of public and pri-vate investments in biomedical R&D are sim-ilarly difficult to distinguish. . . .Not all biomedical innovations have arisenfrom biomedical research and development. . . .The relative contributions of domestic and for-eign investments in biomedical R&D will be-come an increasingly important issue.Assignments of improvement in health to spe-cific biomedical innovations is not always pos-sible. . , . Observed improvements in healthmay have resulted from public health meas-ures or changes in life style and the environ-ment. . . .The economic valuation of improvements inhealth raises important conceptual questions.In particular, innovations that prolong lifegenerally result in increased economic trans-fers from younger, productive generations toolder, less productive generations.

~Charies L. Vehom, et al., “Measuring the Contribution ot Bio-medical Research to the Production of Health, ” Research POIICVvol. 11, 1982, p. 4.; and Mushkin, “Health as an Investment, op.cit., p. 133.

Table 5.—Economic Costs of Disease: Differential Estimates

E s t i m a t e sNIH est imates prov ided to Est imates of developed by methodology of

D i s e a s e H o u s e A p p r o p r i a t i o n s C o m m i t t e ea s p e c i a l C o m m i s s i o n sb S o c i a l S e c u r i t y A d m i n i s t r a t i o nc

Arthritis . . . . . . . . . . . . . . . . . . $4 bi l l ion $13 bi l l ion $3.5 bi l l ionAsthma . . . . . . . . . . . . . . . . . . . $187 mi l l ion $855 mi l l ionBlindness . . . . . . . . . . . . . . . . . $5.2 bi l l ion $2.2 bi l l ionCancer . . . . . . . . . . . . . . . . . . . $15 to $25 billion (range) $17.4 bi l l ion

Diabetes . . . . . . . . . . . . . . . . . . $3.5 bi l l ionDigestive disease . . . . . . . . . . $16.5 bi l l ion $17.5 bi l l ionHeart, lung, and

blood disease . . . . . . . . . . . $58 bi l l ion $12 bi l l ionEpilepsy . . . . . . . . . . . . . . . . . .. $4.3 bi l l ion $522 mi l l ionInfluenza . . . . . . . . . . . . . . . . . $700 mi l l ion $4,1 bi l l ionMental illness . . . . . . . . . . . . . $40 billion $36.8 billion $13.9 bi l l ion

$5.3 billion

aE~t ,mate~ offered , n the Iestl mony Of N I H I nst I tute dl rectors or staff or orovlded I n supplementary mater!als I n the course of hearln9s on N I H a~ ~roo at Ions ‘or

ftscal year 1977bEstlmates devel~ed ~espectlvely, by the National commission on Afihrltls and Related Musculoskeletal Diseases and the Natlonai Cornrnlsslofl on DiabetesC Es~,mates devel~ed ~,th the hel ~ of Barbara S c~per, Off Ice of Research and Statl stlcs, Soctal Security Adm!n! St rat Ion

SOURCE StePhen P Strickland, Research md the Hea/fh of Amertcans (Lexington, MA” Lexington Books, 1978), P 46

22

7. Although considerable attention has been de-voted to valuing loss of life, the state of theart in gauging improvements in quality of lifeis far less advanced.

8 . . . . assessments [based on gains in productiv-ity] may substantially understate the public’swillingness to pay for certain innovations. 29

The three sectors analyzed above have similari-ties that make them amenable to quantitativeanalysis. Federal agricultural R&D is particularlysuited to econometric analysis because the farmersthemselves do almost no research; the governmentplays a significant role in promoting the applica-tion of its research; the government is a major cus-tomer, both directly and through farm and tradepolicies; and productivity improvements are easyto measure. The aviation industry is also a long-established, well-defined sector, dominated by Fed-

“Jeffrey R. Hams, “Biomedical Research and l>evelopment:Measuring the Returns on Inv@ment, ” currently unp lbiished type-script of the National Academy of Sciences, National Academy ofEngineers, and Institute of Medicine, November 1985, and for a fur-ther discussion of the basic-applied distinctions and the pathwaysbetween the two, see Julius H. Comroe and Robert Dripps, “Scien-tific Basis for the Support of Biomedical Science, ” SCience, vol. 192,Apr. 9, 1976, pp. 105-111.

eral support. Aviation R&D is heavily weightedtoward the development end and often incre-mental, making it easier to trace the returns onresearch. Despite the ease of tracking the prod-ucts of the aviation industry and the extensive his-torical records, the case illustrates the difficultyof defining the scope of outputs to be includedin an economic analysis; should these include onlyimproved air transportation services, overall im-proved transportation, or all the indirect socialand economic benefits of the airline industry? Thehealth sector is less tractable, but has received sig-nificant attention because of the large sums of Fed-eral money involved. Many of the great advancesin heath care come serendipitously from basic re-search, making it difficult to trace the return oninvestment.

These analyses do not reveal the extent to whichreturn on private or Federal R&D investment ofan industry depends on variables such as R&Dintensity, size, degree of concentration in a fewlarge companies, whether the industry is emerg-ing or stagnant, extent of technological competi-tion within the industry, capital intensity}’, mar-ket position, and government influences otherthan R&D.

SPINOFFS AND SPILLOVERS: NASA

The bulk of Federal research supports the in-ternal missions of agencies like the Departmentof Defense (DOD) and NASA. While not aimedat commercial products, this research contributesindirectly to the development of commercial prod-ucts or processes. These “spinoffs” and “spillovers”differ from the direct economic impacts of re-search sponsored by the Department of Agricul-ture, NIH and the civil aeronautics program ofNASA in that they are unintended byproducts ofactivities carried out primarily to support non-economic mission agency goals, such as the ex-ploration of space and national security. The eco-nomic impacts of the NASA R&D programs havebeen studied with special thoroughness and willtherefore be described in some detail.

Three different approaches have been used toestimate the economic benefits of NASA’s pro-grams, according to former NASA chief econo-

mist, Henry Hertzfeld.30 First are macroeconomicstudies similar to those described for R&D inves-tments in the economy as a whole. Second aremacroeconomic analyses of the direct and indirectbenefits from inventions and innovations result-ing from the NASA R&D programs. Third arestudies of the patents and licenses resulting fromspace R&D programs, used as a measure of thetransfer of technology to the private sector.

Macroeconomic Studies

The first macroeconomic study, carried out bythe Midwest Research Institute in 1971, estimatedproductivity changes in the national economy and

‘Henry R. Hertzfeld, “Measuring the Economic Impac: ot Fed-eral Research and Development Investments in Civilian Space Activ-ities, ” paper presented to the Nationai Academy of Sciences Work-shop on “The Federal Role m Research and Developer:, ” Nov.21-22, 1985.

23

then subtracted changes due to capital, labor, andsuch non-R&D residuals as demographics, edu-cation, length of workweek, and economies ofscale. The discounted rate of return to NASA-sponsored R&D, calculated by doing a least-squares regression of the remaining residual, cameto 33 percent, or a seven to one benefit/cost ra-tio. This study has several major liabilities. As-signing residual benefits to one factor, such asR&D, is inherently dangerous because we cannotbe certain about the importance of other, un-known residual factors. The study did not lookspecifically at NASA R&D, but simply gave itcredit for a proportionate share of the benefits ofall R&D. The lifetime for benefits was arbitrar-ily set at 18 years .31

Chase Econometrics, in 1975, carried out a farmore sophisticated analysis of the economic im-pact of NASA R&D, again using a residual cal-culation from a production function. Chase cal-culated a “cumulative ‘productivity’ return toNASA R&D of 14 to 1,” which “translated intoan annual discounted rate of return of 43 percentto NASA outlays. ” However, when asked to repli-cate its methodology in 1980, using an “updatedand longer time series, ” Chase found that “produc-tivity changes from NASA R&D spending provednot to be statistically different from zero. ” Hertz-feld concludes that the revised Chase study “showsthat due to the theoretical and data problems withthe macroeconomic model and data sets available,this approach to finding aggregate economic re-turns to R&D expenditure is difficult at best, andprobably impossible. “32

Macroeconomic Studies

The two major macroeconomic studies of theeconomic benefits of NASA R&D programs usedthe “consumer surplus” approach to estimatingthe value to society of introducing a new prod-uct or reducing the cost of an existing product.The “consumer surplus” approach assumes thatmany people would be willing to pay more thanthe market price for a new or improved product,and uses supply and demand curves to estimatethat “surplus. ” Based on this approach, Mathe-matical, Inc., in 197.5, estimated the overall bene-

fits to society from four NASA-stimulated tech-nologies-gas turbine engines, integrated circuits,cryogenics, and “NASTRAN,” an advanced com-puter program dealing with structural analysis.They found that “over a ten year period from 1975to 1984 the four technologies could be expectedto return a discounted total of $7 billion (in con-stant 1975 dollars) in benefits that were attribut-able to NASA’s involvement in their develop-merit. ” This could be compared to a total NASAbudget in fiscal year 1975 of $3.2 billion. (Butabout $30 billion over that 10-year period in 1975.)33

In 1977, another consulting firm, Mathtech,Inc., conducted a benefit/cost analysis of productsadopted by the private sector as a result of NASA’sformal technology transfer program. Mathtechonly estimated the costs to the private sector “offurther developing and transferring the innova-tions rather than the costs of the initial spaceR&D development of the technology. ” The as-sumption here was that the initial developmen-tal costs would have had to be incurred for thespace program whether the technology was trans-ferred or not. Unfortunately, this makes a calcu-lation of the return on the NASA investment im-possible. However, the benefit/cost ratios to theprivate sector were most impressive: 4:1 for thecardiac pacemaker, 41:1 for a laser cataract tool,68:1 for a nickel-zinc battery, 340:1 for zinc-richcoatings, and 10:1 for a human tissue simulator.34

Both the Mathematical and the Mathtech studiesare undermined by important weaknesses in the“consumer surplus” theory. The demand curvesused to calculate the surpluses in that theory areinherently unreliable when applied to new tech-

nologies that have no well- formed demand func-tion and to evolving technologies whose demandfunctions change over time. In addition, bothstudies fail to compare benefits to NASA devel-opment costs.

Patent Analysis

A third approach is to study what industry doeswith the licenses and patent waivers granted byNASA. Analyzing patent waivers, in which NASAallows a company to patent an invention devel -

“Ibid., p. 9.“Ibid., pp. 11-12, emphasis added.

“Ibid., pp. 18-19.“Ibid., pp. 19-20.

24

oped under contract, Hertzfeld has found that thecommercialization rate (total commercialized in-ventions divided by total waivers) averaged 20.8percent over the period 1961-75, with electricalmachinery, communications equipment, and in-struments accounting for over 69 percent of allcommercialized specific waivers. Of the more than197 NASA patents licensed to industry, Hertzfeldfound that 54 were commercialized between 1959and 1979. This was still a very small fraction (1. spercent) of the more than 3,500 patents ownedby NASA at the time.35

Finally, Hertzfeld points out that another eco-nomic benefit of the NASA space R&D programhas been the creation of a multibillion dollar sat-ellite communications industry and a tenfold re-duction in the cost of satellite communications.However, Hertzfeld stresses that “economic re-turns are not the primary reason for space invest-merits, ” and therefore “no economic measure orcalculation can, by definition, encompass the en-tirety of the return to space investment. ”36

“Ibid., pp. 24-26.36Ibid., p. 4.

IMPLICATIONS FOR USING ECONOMIC INVESTMENT MODELSTO GUIDE FEDERAL R&D DECISIONMAKING

The studies described above present a discour-aging picture for the use of economic returns asa valid measure of the value or desirability of Fed-eral research Funding. Although strong positivereturns to private sector research funding in gen-eral, and basic research funding in particular, havebeen indicated by macro-level econometric stud-ies, no such positive returns have been shown forFederal research spending. Using econometricmodels to estimate the aggregate rate of returnto all Federal research pushes the methodologybeyond its limited capabilities.

The fundamental stumbling block to placing aneconomic value on Federal R&D is that improvingproductivity or producing an economic return isnot the primary justification for most FederalR&D programs. The basic justification for Fed-eral support of R&D is to encourage research thatis socially desirable, high risk, or in the nationalinterest but that is unlikely to be funded by theprofit-driven private sector. The very concept ofmeasuring the return to Federal investment in re-search in economic terms is therefore inherentlyflawed.

Economists who have studied this issue describethese flaws must vividly. At a fundamental level,Federal research is a public good which cannotbe valued easily in economic terms. As Peter Reisshas expressed it:

Typically the [activities of] the Federal govern-ment . . . produce things that have no marketvalues that economists can even begin to mea;-ure. There is no market price, for example, formost health advances, and there is no conceiva-ble evaluation for . . . public goods . . . like astrong national defense. These things are just notquantifiable .37

In many cases substantial economic benefits,such as NASA spinoffs and the computer andmicroelectronics technology spawned by DOD re-search, were secondary to the primary politicaland national security missions.

Frank Lichtenberg has found that Federal pro-curement has a far greater and more positive ef-fect on private R&D expenditures than does Fed-eral R&D. This finding is consistent with that ofa number of other economists. 38 The direct influ-ences of R&D support keep company with indirectbut powerful Federal influences on private sectorR&D through patent law, macroeconomic pol-icies, tax incentives, trade policies, technologytransfer, antitrust practices, and regulation.

‘-The National Academy of Sciences, Committee on Science,Engineering and Public Policy, The Federal Role [n Research andDevelopment, typescript transcript (Washington, DC: NAS, Nov21, 1985), p. .s3.

~Nelson, op. cit., pp. 459462, 471-472.

25

In addition, there are fundamental flaws withthe econometric methodology for measuring re-turns on investment when applied to FederalR&D. First, macroeconomic studies measure theaggregate return to the total expenditure on pastR&D. They do not provide any information onthe incremental return to the marginal expendi-ture on future R&D, which is the concern of thepolicymaker. Another major stumbling block ineconometric analyses is that they measure inputs(R&D investments) and outputs (productivitychanges) while ignoring the process that goes onin between. That process is the critical stage ofturning laboratory research into a tangible return—innovation and commercialization. Researchcannot result in product or process improvementunless each step in the move from idea to marketis successful: advanced development, pilot studies,legal blessing of patents and licenses, production,and marketing. These intermediate commerciali-zation steps are as important a factor in the movefrom R&D to productivity as is the R&D invest-ment itself. However, the Federal Government hasdirect control only over its R&D investment, andvery little influence over commercialization in theprivate sector.

According to Hertzfeld, the production func-tion model used in most econometric analyses “as-sumes that a formal relationship between R&Dexpenditures and productivity exists” but “skipsa number of steps in the process . . . creation ofnew knowledge, which will lead to ideas, inven-tions, innovations, and eventually, with propermarketing and distribution, commercial products. . . Assuming that all of these missing steps takeplace from any given set of R&D expenditures istaking a giant leap without looking. 39

Hertzfeld concludes his study of the economicimpact of Federal R&D investments in civilianspace activities with a statement that seems appli-cable to the more than 80 percent of the Federalresearch budget that is not aimed directly at im-proving economic competitiveness:

. . no economic study should attempt to put a“bottom line” ratio or return on space R&D in-vestments. There is no such number in existence—it only lives in the uncharted world of general

‘Hertzfeld, op. clt,, p, 7.

equilibrium theory. . . . All such numbers areproducts of economic models with many limitingassumptions. Even when these assumptions andqualifications have been carefully laid out, the ex-istence of the number is an attractive bait to thosepoliticians and others who need to justify spaceR&D. Once a “total” returns number is used, itquickly finds its way into misuse .40

Clearly, R&D expenditures may be conceptu-alized as an investment flow, largely on the no-tion that R&D expenditures support the growthof an “R&D capital stock” or knowledge base,which contributes to economic growth and pro-ductivity improvement. However, a number ofscholars recently have argued that the overenthu-siastic application to investment decisionmakingof principles of portfolio management has led U.S.firms to underinvest in new technologies becauseof an “extreme of caution” (see ch. 4). Hence, evenif satisfactory methodologies to calculate returncould be developed, it might be unwise to adoptthem.

R&D investment, and investment in basic re-search in particular, are characterized by highlevels of uncertainty about their outcomes. A mar-ket that does not yet exist cannot be measured.While quantitative models of financial and otherinvestment planning do provide methods for re-ducing risk, there exist virtually no models thatcan incorporate uncertainty. An explanation ofthis problem requires a brief description of thedifferences between risk and uncertainty.

Risk exists when decisionmakers can define therange of possible outcomes and assign probabil-ities to each. Models of risk analysis and reduc-tion that have been developed for financial andinvestment decisionmaking rely heavily on theability to quantify risk by assigning a probabil-ity to a specified set of likely, discrete outcomes.

*Ibid., p. 42, emphasis added.41 Spence’s analysis of “Investment, Strategy and Growth in a

New Market” explicitly disavows any attempt to deal with uncer-tainty:

And finally, and less happdy, uncertainty about demand, tee)moloW,the rates of entry, and competitors’ behawor—a]l of which are practicalproblems for firms and inherent features of most new markets—is setaside to tocus on the Issue of the opt]mal penetration ot the market andthe dynamic aspects of strategic Interact Ion Integrating uncertainty, intoan appropriate model rema]ns a high prtor]ty research topic

A.M. Spence, “Investment Strategy and Growth In a New hlar-ket, ” 5efl,/oumaf of EconornIcs, vol. 10, No. 1, 197Q, pp. 1-19 (quo-tat]on from p. 2).

26

True uncertainty, as defined by Frank H. Knightand others,42 is the inability to specify possibleoutcomes. Estimating consequences, and thereforerisks, is impossible. If one cannot specify out-comes, possibly due to the uniqueness of the proc-ess (e.g., the creation of an entirely new technol-

ogy) or the lack of historical data on relationsbetween actions and outcomes, quantitative mod-els for risk-minimizing investment strategies arenot applicable.

R&D investment, especially basic research in-vestment, is a classic example of investment un-dertaken under conditions of severe uncertainty.Not only are research outcomes dimly perceived,

‘Nelson and Winter describe the search behavior of a firm in anuncertain envirorunent as follows:

The areas surveyed by a decisionmaker inside the firm may well in-clude identifiable “alternatives” that could be explored, but these are onlydimly perceived and it may not be at all clear which will turn out tobe best. The process of exploring perceived alternatives, or ● xogenous● vents, may bring to tight other ahematnws not even contemplated inthe original assessments. . It is clearly inappropriate to apply uncrit-ically, in the analyt:ca[ treatments of that process, formalisms that posita sharply defined set of altematlves. .

Richard R. Nelson mid Sidney G. Winter, An Evolutionary The-ory of Economic Change (Cambridge, MA: Harvard UniversityPress, 1982), pp. 171-172.

assigning credible probabilities to the possible out-comes is impossible. Quantitative models devel-oped to assess risk in energy exploration or finan-cial management cannot address the uncertaintyinherent in basic research spending decisions. Asone moves from basic research to applied researchor development, quantifiable risk replaces uncer-tainty. For this reason, quantitative models arelikely to be far more useful in the evaluation ofapplied R&D or development decisions than forexploratory research. In addition, quantitativemodels are more applicable to decisions about dis-tributing R&D investments among research instal-lations or performers within a single disciplinethan allocating basic research funds across com-peting fields.

For the wealth of reasons presented above, itis clear that using economic returns to measurethe value of specific or general Federal researchexpenditures is an inherently flawed approach.The only exceptions to this rule are certain Fed-eral R&D programs whose specific goals are toimprove the productivity y of particular industriesor industrial processes.

Chapter 3

Noneconomic QuantitativeMeasures: The “Output”

of Science

chapter 3

Noneconomic Quantitative Measures:The “0utput” of Science

Having identified severe drawbacks to the useof econometric models to evaluate Federal R&D,OTA looked elsewhere for objective quantitativemeasures. The only quantitative approach to theevaluation of research output is bibliometrics,which analyzes scholarly publications for indica-tions of quantity and quality. The underlying as-sumption of this approach is that knowledge isthe essential product of research and publicationsare the most readily identified manifestations ofthat knowledge. With a gradually evolving meth-odology, bibliometricians have attempted tomeasure objectively the quantity and quality ofresearch results. They have achieved some suc-cess in comparing projects within a discipline, andless in comparing disciplines. Bibliometric anal-ysis does not address the most important policyquestion: how to compare the value of Federalresearch with other Federal programs.

BIB BIOMETRICS

The quantitative analysis of scientific publica-tions is in its second generation. The first gener-ation, spurred by Eugene Garfield’s founding ofthe Science Citation Index and Derek de SollaPrice’s efforts, ’ explored the feasibility of under-standing science through its literature alone. Priceboldly named this approach the “science of sci-

For a first person retrospective, see Eugene Garfield, Essays ofan Information Scientist, vol. 1, 1962-73; vol. 2, 1974-76 (Philadel-phia, PA: ISI Press, 1977). For examples, see Eugene Garfield, etal., The Use of Citation Data in Writing the History of Science (Phil-adelphia, PA: Institute for Scientific Information, 1964); Derek deSolla Price, “Networks of Scientific Papers,” Science, vol. 149, July30, 1965, pp. 510-515; Derek de Solla Price, “Is Technology His-torically independent of Science? A Study in Statistical Historiog-raphy, ” Technology and Culture, vol. 6, fall 1965, pp. 553-568.

A considerable amount of quantitative infor-mation about the U.S. science and engineeringenterprise is published regularly by the NationalScience Foundation (NSF), the National Institutesof Health (NIH), and the National Research Coun-cil (NRC) in their reports on funding, personnel,degree attainment and graduate education. Every2 years NSF publishes a 300-page compilation ofthis information, Science Indicators. Science in-dicators could be used to provide a rough meas-ure of the health of the research enterprise in theUnited States if some agreement could be achievedin the science policy community about which ofthe thousands of numbers published by NSF aremost relevant to that task. The use of science in-dicators to measure the quality of the researchprocess in the United States is discussed later inthis chapter.

ence” and published demonstrations of its heuris-tic, if not immediate policy, value, z

The second generation, now a decade old, soughtto develop and exploit publication and citationdata as a tool for informing decisionmakers, espe-cially in Federal agencies and universities.3 Thiscurrent generation has many of the features 01 an

‘Derek de Solla Price, Littfe Science, Big %ertce I FJew YorkColumbia Umverslty Press, 1963).

‘For reviews, see Yehuda Elkana, et al. (eds. ), Toavard A .Wet-ric of Science: The Advent of Science Indicators ( New York. JohnWiley & Sons, 1978); Francis Nann, “Objectivity Versus Relevancein Studies of Scientific Advance, ” Scientometr~cs, vol. 1, Septem-ber 1978, pp. 35-41. The use of pro]ected citation data in a con-troversial promotion and tenure case is described in N.L. Geller,et al., “Lifetime-Citation Rates to Compare Sc]ent]sts Work, ” So-cial Science Research, VOI. 7, 1978, pp. 345-305

29

3 0

institutionalized scientific specialty: multidisciplin-ary journals and practitioners, a clientele (bothconsumers and patrons), and numerous claims tothe efficacy of “bibliometrics” as a policy tool.4

The quantitative analysis of scientific publicationshas arguably established its place in the evalua-tion of research outcomes and as an input bothto the allocation of resources for research and tothe expectation that the growth of scientificknowledge can be measured, interpreted, and in-deed, manipulated.

This chapter focuses on the second generationof noneconomic quantitative measures of scien-tific research results and evaluates its usefulnessto policymakers. The chapter assesses the mostpromising approaches and methods that havebeen employed and suggests how quantitativedata and models could be refined to augment deci-sionmaking processes in science.

The First Generation of Bibliometrics(1961-74)

The pioneers of bibliometrics searched for waysto understand science independent of the scien-tists themselves. First-person accounts, question-naires, and historical narratives all require someform of cooperation or consent of the scientistsinvolved. This dependence on self-interest sourcescould bias the results. Bibliometric pioneers of theearly 1960s saw a need first to reconstruct, then

‘Cofounded in 1978 by Garfield and Price, .f%entometrics be-came the flagship journal of bibliometncs. Its contributors seem tocome pmmarily from information science, psychology, and sociol-ogy. Other spurs to the institutionalization and visibility of bibllo-metrics has been, since 1972, the National Science Board’s biennialScience Indicators series and the ongoing work of the Institute forScientific Information (especlaily Henry Small) and Francis NarmsComputer Horizons, Inc. (discussed below). For historical perspec-tives on the development of this specialty, see Daryl E. Chubin, “Be-yond Invisible Colleges: Inspirations and Aspirations of Post-1972Social Studies of Science, ” %ientometrics, vol. 6, 1985, pp. 221-254; Da~l E. Chubin and S. Restive, “The ‘Mooting’ of ScienceStudies: Strong Programs and Science Policy,” in K.D. Knorr-Cetinaand M. Mulkay (eds. ), Science Observed (London and Beverly Hills,CA: sage, 1983), pp. 58-83. Also see the special issue of sClW’ttO-

metn”cs, vol. 6, 1985, dedicated to the memory of Derek Price.

to monitor and predict, the structure and prod-ucts of science. Eugene Garfield and Derek deSolla Price talked about “invisible colleges” andthe tracing of “intellectual influence” as a mirrorheld up to science, imperfect but public, using theformal communication system of science. Scienceliterature could be studied—without recourse tothe authors—to open new vistas, both practicaland analytical, once it was cataloged, indexedand made retrievable.

With the creation of the Science Citation Index(SCI), the scientific literature became a data sourcefor the quantitative analysis of science. It gener-ated both the concepts and measurement tech-niques that formed the bedrock of bibliometrics. 5

These include the principal units of analysis: pub-lications (papers, articles, journals), citations (bib-liographic references), and their producers (indi-vidual authors and collaborators in teams). Whensubjected to the primary methods of analysis—counting, linking, and mapping—these units yieldmeasures of higher order concepts: coherent so-cial groups, theory groups, networks, clusters,problem domains, specialties, subfields, andfields.

Computers aided the increasingly sophisticatedmanipulation of documents in the growing SCIdatabase. Journal publications could be countedby author, but also aggregated into schools of

These are touted , debated , and assa i led In Daryl E. ChubIn,“The Conceptualization of Sclent]fic Specialties, ‘ The Soc~o/og~calQuarterly, vol. 17, autumn 1976, pp. 448-476; Daryl E. Chub]n,“Constructing and Reconstructing Scientific Reality: A Meta-Analysis, ” ]ntemationa] Society for the Sociology ot Know]edg*Newsletter, vol. 7, May 1981, pp. 22-28; Susan E. Cozzens, “Tak-ing the Measure of Science: A Rewew of Citat]on Theories, ” ISSKNewsletter, vol. 7, May 1981, pp. 16-21; D. Edge, “QuantitativeMeasures of Communication in Science: A CriticaI Review, ” His-to~ of Science vol. 17, 1979, pp. 102-134; and in various chaptersin Elkana, et al., op. cit.

thought or whole institutions. b Consistent and in-fluential contributors to the literature could beidentified by co-citations (the number of times twopapers are cited in the same article) and separatedfrom occasional authors. The resultant co-citationclusters could be depicted as a “map of science”for a given year showing the strength of linkswithin clusters and the relations, if any, amongthem. 7

By the mid-1970s, bibliometricians were con-structing structural and graphical maps of the do-mains and levels of research activity in science.Further, they were comparing these pictures toother accounts, built on biographic and demo-graphic information, informal communication,and other informant-centered data, to depict howresearch communities—their research foci, intel-lectual leaders, and specialized journals-changeover time. They thus offered a more comprehen-sive perspective on the growth of knowledge, atleast in terms of its outputs, than was ever previ-ously available. ” Analysts differed in their inter-pretation and application of the data, and the life

‘Semmal work here is D. Crane, Invisible Colleges: Diffusion ofKnowkdge in Sa’enti”c Communities (Chicago and London: Univer-sity of Chicago Press, 1972); B.C. Griffith and 14. C. Mullins, “Co-herent SociaI Groups in Scientific Change, ” Sc]ence vol. 177, Sept.15, 1972, pp. 959-964; N.C. Mullins, ‘The Development of a Scien-tific Specialty: The Phage Group and the Origins of Molecular Bi-ology, ” Minerva, vol. 10, 1972, pp. 52-82; N.C. Mullins, Theoryand Theory Groups in Contemporary American Sociology (NewYork: Harper Row, 1973); and Garfield, op. cit., “Corporate in-dex” that lists publications by institution of author.

‘The methodological groundwork for co-citation analysis is pre-sented m B.C. Griffith, et aI., “The Structure of Scientific Litera-tures II: Toward a Macro- and Microstructure for Science, ” .SaenceStudies, vol. 4, 1974, pp. 339-365; H.G. Small, “Co-citation in theScientific Literature: A New Measure of the Relationship BetweenTwo Documents, ” -loumal of the American Soc~ety for informa-tion Science, VOI. 24, 1973, pp. 265-269; H.G, Sma]], “Multiple Ci-tation Patterns m Scientific Literature: The Circle and Hill Models, ”Information Storage and Retrieval, vol. 10, 1974, pp. 393-402; H.G.Small and B.C. Griffith, ‘The Structure of Scientific Literatures I:Ident@ing and Graphing Specialties,” Science Studies, vol. 4, 1974,pp. 1740.

‘Noteworthy illustmtions are discussed in Daryl E. Chubin,‘The Conceptualization of Scientific Specialties, ” op. cit., and G.N.Gilbert, “Measuring the Growth of Science: A Review of indica-tors of Scientific Growth, ” Sa”entometrics, vol. 1, September 1978,pp. 9-34.

of the community responsible for the outputstended to remain unobserved. Nevertheless, bib-liometric analysis began to offer the promise ofthe independent baseline implied in Price’s phrase“the science of science. ”9

The Second Generation (1975=85)

The legacy of the first generation was the prom-ise of its scholarly literature. The second genera-tion has attempted to deliver on the promise thatbibliometric analysis could be predictive and relia-ble for decisionmaking. That promise has yet tobe fulfilled, for reasons that will be discussed be-low. However, there is growing evidence that thequantitative assessment of science warrants theattention it is now receiving from policymakersboth in the United States and Europe.

The analysts discussed below have used biblio-metrics to anticipate the source of “greatest con-tributions” and identify promising research proj-ects. They produce policy-relevant documents andrecognize intervention decisions as a desirableconsequence of their work. Several governmentshave funded their efforts. A look at the leading

‘For example, m 1%9, Price’s “Measuring the Size of Science, ”Proc&”ngs of the Israel Academy of Sciences and Humanltzes, vol.4, 1969, pp. 98-111, tied national pubiicat]on actlvltv to percent otGNP allotted to R&D. By 1975, F. Narin and M. Carpenter f“Na-tional Publication and Citation Comparisons, ” IASIS, voi, 25, pp.80-93) were computing shares, on a nation-bv-nation basis, ot theworld literature, and characterizing interrelations among ]ournals(Francis Nann, et ai., “Interrelationship of Scwntlflc Joumais, ‘]ASISvol. 23, 1972, pp. 323-331 ), as well as the content of the ]lteratureIn broad fieids (Francis Narin, et al., “Structure of the BiomedlcaiLiterature, ” JASZS, vol. 27, 1976, pp 25-45). These analyses em-pioyed algorithms for tailymg, welghlng, and llnklng keywords Inarticle titles to citations aggregated to ]ournals and authors nation -ot-affiiiat]on at the time of publication. Some wouid call this meth-odology “crude”; others would herald Its ‘sophlstlcat]on tor dls-cermng patterns in an otherwise massive and perplexing literatureThe latter is preciseiy the mentality guiding the Sc]ence ind~catorsvolumes and foreshadowed in two other ploneenng papers ot thefirst generation: Eugene Garfield, “Citation Indexing tor StudyingScience, ” Nature, vol. 227, 1970, pp. 659-671; Derek de Soiia Price,“Citation Measures of Hard Saence, Soft Science, Technology andNon-science, ” Communication Among Scientists and Engrneers,C. Nelson and D. Poilock (eds.) (Lexington, LIA: DC, Heath, 1970),pp. 3-22.

32

practitioners will reveal how they approach thequestion:

How can we characterize the effects of decisionsabout funding programs as they reverberate intothe various levels of the scientific community: upfrom “fields” into disciplines and down from“fields” into research areas or teams?10

Francis Narin of Computer Horizons, Inc., isthe veteran performer, linking the two genera-tions. His computerized approach is based on thecomponents of the Science Citation Index andused in conjunction with other data, such as theNational Library of Medicine’s MEDLINE and theNIH in-house grant profile system, Informationfor Management Planning, Analysis, and Coordi-nation (IMPAC). Although Narin’s work tends

toward the macroscopic, its manipulations havegrown more sophisticated in their capability ofaddressing micro-level questions. Narin’s meth-odology answers quantitatively the followingkinds of questions:

• Are articles published in basic journals refer-enced in clinical and practitioner journals[these types derive from Narin’s own classifi-cation of article content in journals]?

● Is there a relationship between priority scoreson research applications and number of arti-cles produced and citations received?

● Are grants to medical schools more produc-tive than grants to academic departments?

• Are young researchers more productive thanolder researchers?

● Is the return on investment mechanism [in-vestigator-initiated proposals] more produc-tive than other support mechanisms?

● How often do National Institute on DrugAbuse, National Institute on Alcohol Abuseand Alcoholism, and other NIH-supportedresearchers cite work supported by the Na-tional Institute of Mental Health?

A common criticism of Narin’s work is that itis too descriptive and relies on ad hoc explana-tion for the observed patterns and trends. Somefeel it is excessively dependent on a literature base-line and does not reflect an understanding of the

‘Susan E. Cozzens, “Editor’s Introduction, ” m “Funding andKnowledge Growth, ” Theme Section, Soaaf Studies o[Science, vol.16, February 1986, forthcoming (quote from mlmeo version, p. 9).

sciences it appraises. In a current project spon-sored by the National Cancer Institute (NCI), “AnAssessment of the Factors Affecting Critical Can-cer Research Findings, ” Narin consciously tries toremedy the problem by working closely from theoutset with a panel of cancer researchers. He istracing key events through participant consensus,the historical record, and various bibliometric in-dicators. Discrepancies are apparently negotiatedas the project unfolds, though the exact negotia-tion procedure is not specified .11

Another departure for Narin stems from his ac-quisition and computerization of U.S. Patent Of-fice case files that will permit mapping of litera-ture citations in patents at the national, industry,and inventor levels. An infant literature has crys-tallized around the notion of “technology indica-tors” with patents signifying the conversion ofknowledge into an innovation with commercialand social value—another tangible return on in-vestment. 12

Irvine and Martin’s (Science Policy ResearchUnit, University of Sussex, UK) evaluation pro-gram in “converging partial indicators” has gainedattention for three important reasons:

1.

2.

3.

They claim to assess the basic research per-formance of large technologydependent facil-ities, such as the European Organizations forNuclear Research (CERN) accelerator and theIsaac Newton Telescope.They have made cost-effectiveness the centralperformance criterion in their input-outputscheme.Their “triangulation” methodology is an im-pressive codification of many separate proce-

“The objective of this project IS to estimate knowledge returnsfrom the U.S. war on cancer. What has been the extent and charac-ter of NCI funding in the cancer literature: are highly cited papersand authors supported by NC1 grants and contracts? More on thi:genre of study is presented below and in other chapters of this tech-nical memorandum, but see Francis Narin and R.T. Shapiro, “TheExtramural Role of the NIH as a Research Support Agency, ” Fed-eration Proceeding, vol. 36, October 1977, pp. 2470-2475.

“M.P. Carpenter, et al., “Citation Rates to Technologically Im-portant Patents, ” World Patent Information, vol. 3, 1981, pp. 161-163; and various case study reports on patent actwity emanatingfrom Battelle’s Pacific Northwest Laboratories, for example, R.S.Campbell and L.O. Levine, Technology indicators Based on Cita-t~on Data: Three Case Studies, Phase II Report prepared for the Na-tional Science Foundation Grant, PRA 78-20321 and Contract2311103578, May 1984,

33

dures and measures that have been advocatedboth by policymakers and analysts. ’3

The synopsis presented below is based primarilyon a review of Irvine and Martin’s articles, fourcritiques, and a reply. 14 The I rv ine a n d M a r t i n

r a t i o n a l e f o r d e v e l o p i n g i n d i c a t o r s o f p a s t r e -search performance is to provide “a means to keep

the peer-review system ‘honest. ’ “ Irvine and Mar-tin caution us further “to distinguish between con-

v e n t i o n a l p e e r - r e v i e w ( i n v o l v i n g a s m a l l n u m b e r

of referees or ‘experts’ on a panel) and our exten-

sive peer-evaluations drawing in very large num-

bers of researchers across different countries and

b a s e d o n s t r u c t u r e d c o n f i d e n t i a l i n t e r v i e w s a n d

attitude surveys. ” 15 For the two investigators, con-

vent iona l grants or journa l peer rev iew i s but a

s ing le ind ica tor ; when combined wi th b ib l iomet -

r i c d a t a o n r e s e a r c h p e r f o r m a n c e a n d e x t e r n a l

assessments o f the l ike ly fu ture per formance o f

n e w f a c i l i t i e s , a s e r i e s o f m u l t i p l e i n d i c a t o r s i s

f o r m e d . I f t h e i n d i c a t o r s c o n v e r g e , I r v i n e a n d

Martin regard the evaluation results as relatively

reliable. 1 6

As proxies, partial indicators must stand for al o t t h a t g o e s u n m e a s u r e d — b y c h o i c e o r o t h e r -

wise . Somet imes the in terpre t ive burden i s over -

whelming ( see tab le 6 ) . No mat ter how sys tem-

a t i c , quant i ta t ive , and convergent the i r f ind ings

appear , I rv ine and Mar t in ’ s use o f t r i angula t ion

is prob lemat ic , as they admit ( see tab le 7 for as u m m a r y ) :

“J. Irvine and B.R. Martin, Foresight in Science: Picking theWinners (London: Frances Pinter, 1984). See especially B.R. Mar-tin and J. Irvine, “Assessing Basic Research: Some Partial Indica-tors of Scientific Progress m Radio Astronomy, ” Research Policy,VOi. 12, 1983, pp. 61-90.

“The five components are: J. Knge and D. Pestre, “A Crltlqueof Inine and Martin’s Methodology for Evaluating Big Science, ”Social Studies of Science, vol. 15, 1985, pp. 425-539; H.F. Moedand A.F.J. van Ram, “Critical Remarks on tie and Martin’s Meth-odology for Evaluating Scientific Performance, ” Social Studies ofS~”ence, vol. 15, 1985, pp. 539-547; R. Bud, ‘T’he Case of the Dis-appearing Caveat: A Critique of Iwine and Martin’s Methodology, ”%cial Studies of Science, vol. 15, 1985, pp. 548-553; H.M. Col-lins, ‘The Possibilities of Science Policy, ” Sm”al Studies of Science,VOI. 15, 1985, pp. 554-558; and B.R. Martin and J. Irvine, “Evalu-ating the Evaluators: A Reply to Our Critics, ” Social Studies ofScience, vol. 15, 1985, pp. 558-575. For brevity, quotes from thecritics will be noted in the text by (page number) only, those fromMartin and Ir-wne as (I!kl, page number).

‘Ibid., p. 566.“Ibid., p. 527.

The fact that the indicators converge in a givencase does not “prove” that the results are 100 per-cent certain—the indicators may all be “wrong”together. However, if a research facility like theLick 3-meter telescope produces a comparativelylarge publication output at fairly low cost, if thosepapers are relatively highly cited, . . . and if largenumbers of astronomers rate it highly in thecourse of structured interviews, we would placemore credibility on the resulting conclusion thatthis was a successful facility than if the same find-ing were arrived at by a panel of three or four“experts” without access to the systematic infor-mation that we have collected. 17

If the output measures do not converge, theresults become quite problematic. There is nostraightforward means of resolution except intu-ition and judgment.

Tables 7, 8, and 9 present typical samples ofthe information one can obtain from Irvine andMartin’s analyses. Table 7 shows for four differ-ent but comparable optical telescopes the aver-age number of papers published per year over thedecade 1969-78, the cost per paper, the numberof citations to work done on that telescope overthe 4-year period 1974-78, the average number ofcitations per paper, and the number of paperscited 12 or more times. This table was part of apaper that demonstrated that the Isaac NewtonTelescope (INT) in Great Britain was more costlyand less productive than several comparable fa-cilities. (This was largely due to a political deci-sion to locate the INT at a poor observing site onBritish soil. Subsequently, it was moved to a morefavorable site at La Palma. ) The table comparesthe various facilities in terms of output (papersper year), cost-effectiveness (cost per paper), in-fluence (citations), and significance of scientificwork (citations per paper and number of paperscited more than 12 times).

Table 8 presents similar output data for worldexperimental high energy physics facilities from1977 to 1980. The table shows, for example, thatalthough the largest number of papers were pro-duced at the CERN proton synchrotrons in 1978,this facility did not have the greatest influence interms of the number of citations to work donethere, nor was it producing the most significant

‘“Ibid., p. 568

34

Table 6.—Main Problems With the Various Partial Indicators of Scientific Progress and Details ofHow Their Effects May Be Minimized

Partial indicatorbased on Problem How effects may be minimized

A. Publication 1. Each publication does not make an equalcounts contribution to scientific knowledge

2. Variation of publication rates withspecialty and institutional context

B. Citation 1. Technical limitations with Scienceanalysis Citation Index:

a. first-author only listedb. variations in namesc. authors with identical namesd. clerical errorse. incomplete coverage of journals 1

2. Variation of citation rate during lifetime ofa paper—unrecognized advances on theone hand, and integration of basic ideason the other

3. Critical citations4. “Halo effect” citations

1

5. Variation of citation rate with types ofpaper and specialty

6. Self-citation and “in-house” citation(SC and IHC)

1. Perceived implication of results for owncenter and competitors may affectevaluation

2. Individuals evaluate scientific contribu-tions in relation to their own (very differ-ent) cognitive and social locations.

3. “Conformist” assessments (e.g., “halo ef-fect”) accentuated by lack of knowledgeof contributions of different centers I

C. Peer

Use citations to indicate average impact of agroup’s publications, and to identify very highlyc i t ed pape rs

C h o o s e m a t c h e d g r o u p s p r o d u c i n g s i m i l a r t y p e sof papers wi th in a s ing le spec ia l ty

Not a problem for research groups

Check manually

Not a serious problem for “Big Science”

Not a problem if citations are regarded as an lndlca-tor of impact, rather than quality or importance

Choose matched groups producing similar typesof papers within a single specialty

Check empirically and adjust results if the inci-dence of SC or IHC varies between groups

1. Use a complete sample, or a large represen-tative sample (25°/0 or more)

2. Use verbal rather than written survey so canpress evaluator if a divergence between expressed opinions and actual views is suspected

3. Assure evaluators of confidentiality4. Check for systematic variations between differ-

ent groups of evaluators!

SOURCE: B.R. Madin and J. Irwne, “Assessing BaaIc Research: Some Partial indicators of Scosntlflc Progress In Radio Astronomy, ” Research Poltcy, VOI 12, 1983

Table 7.—Output indicators for OpticalTelescopes —A Summary

Lick KPNOa CTIOb INTc

3-meter Z. I-meter 1 ,5-meter 2,5-meter

Average number of paperspa., 1969-78

Cost per paper m 1978Citations to work of past 4

years m 1978 . . .Average citations per paper

in 1978, ., .,Number of papers cited 12

or more times in a year,1 9 6 9 - 7 8 , .

42 43 35 7ƒ13k ƒ 7 k ƒ 6 k ƒ 6 3 k

920 710 580 140

4.2 3.3 3.3 3.6

41 31 21 4%itt Peak National Observatory (U.S.).bcem Tololo lnterAmeflc~ observatory (Chile).%MUIC Newton Telescope (Great BdteW.

SOURCE: B.R. Martin and J. Irvine, “Evaluating the Evaluators: A Reply to OurCritics, ” Soc/a/ Studies of Science, vol. 15, 1985, p. 569.

scientific work in terms of average citations perpaper or number of highly cited papers. One canalso see the decreasing importance of the CERNproton synchrotrons as newer machines such asthe CERN super proton synchrotrons and the Gern-man Electron Synchrotrons Laboratories (DESY)accelerator at Hamburg come on-line and beginto produce important results.

Table 9 presents the high energy physicists’ ownevaluations of the relative contributions of thedifferent facilities described in table 8, based ona mail survey of 182 researchers in 11 countries.These evaluations are based on the relative out-puts of the different accelerators over their entire

35

Table 8.—Experimental High-Energy Physics, 1977-80

Percent of papers Percent ofpublished in past citations to work Average c i ta t ions Highly cited papers:

2 years of past 4 years per paper number cited in times

1978 1980 1978 1980 1978 1980 n > 1 5 n > 3 0 n > 5 0 n > 1 0 0

CERN proton synchroton . . . 22.0°/0 11.5 % 14.50/0 12.50/o 2.2 2.2 13 2 1 0Brookhaven/AGS . . . . . . . . . 5.50/0 5 . 5 % 5.0 ”/0 3 . 0 % 2.7 1.6 0 0 0 0Serpukhov. . . . . . . . . . . . . . . 12.0°/0 14.00/0 4 . 0 % 5.0 0/0 1.2 1.2 0 0 0 0CERN ISRa . . . . . . . . . . . . . . 4 . 5 % 5.5 % 7 . 0 % 7 . 5 % 5.4 4.4 11 2 0 0Fermilab . . . . . . . . . . . . . . . . 16.5°/0 1 9.0% 3 2 . 0 % 2 1 . 5 % 7.3 3.6 40 10 5 1CERN super proton

synchroton . . . . . . . . . . . 2.50/o 8.50/o 4 . 0 % 8.50/o 12.7 5.0 19 7 3 0SLAC b . . . . . . . . . . . . . . . . . . 9 . 5 % 6.00/0 1 5 . 0 % 1 1 . 5 % 5,7 4.4 26 6 1 1DESY . . . . . . . . . . . . . . . . . . . 4.0 ”/0 6.50/o 5 . 5 % 15.50/0 5.7 8.8 36 16 4 0Rest of world . . . . . . . . . . . . 23.5°/0 24.00/o 13.0 ”/0 15.00/0 2.0 1.9 19 5 0 0World total . . . . . . . . . . . . . . 1,115 930 8,190 5,090 3.5 3.0 164 48 14 2

100’%0 1 0 0 % 1 0 0 % 1 0 0 %alritwswtlng storage rlrias.bstantord Ltnear Accelerator Center.

SOURCE J Irvine and B R Martin, “Quantitative Science Policy Research, ” testimony to the House Committee on Science and Technology Oct 30 1985

Table 9.—Assessments (on a 10-point scalea) ofMain Proton Accelerators in Terms of “Discoveries”

and “Providing More Precise Measurements”

Overall rankingsSelf -rankings Peer-rankings (sample size= 169)

Discoveries:Brookhaven/AGS 9.5( ± 0. 1 ) 9.0( ± 0.1) 9.2( ± 0. 1 )CERN PS 7 1(± 0.2) 6.7(± 0.2) 6.9( ± 0. 1 )CERN ISR 6.8( ± 0. 3) 5.9(± 3.2) 6.1(±0.2)CERN SPS 5.9(±0.3) 5.6(±0.2) 5.7( ±0. 1 )F e r m i l a b 7 4( ± 0.3) 7.1 ( ±0. 1 ) 7.2( ±0.1 )Se rpukhov 3.8( ±0.5) 2.6( ±0.1 ) 2.7( ±0. 1 )

More precise measurements:Brookhaven/AGS 7 1( ±0.2) 7,2(±0.2) 7 2( *o. 1 )CERN PS 8.5( ±0. 1 ) 8.5( ±0.l ) 8.5( ±0.. 1 )CERN ISR 7 3( *0.3) 6.9( ±0..2) 7 0( ±0. 1 )CERN SPS 8 2( ±0.2) 8.2( *0.2) 8.2( ±0. 1)Fermilab 6.3( *0.2) 6.0( ±0. 2) 6. 1( ±0..1 )Serpukhov 4.3( ±0.5) 3.5( *0.2) 3.6( ±0..2)a10-top. The assessements are based on the relative outputs from the accelera-tors over their entire operational careers up to the time of the !nterviews withhlgh-energy physicists in late 1981 to early 1982.

SOURCE J Irvine and B.R Marf!n, “Quantltat!ve Science Policy Research, ‘ tes-tlrc >ny to the Iiousa Commtttee on Sctence and Technology, Oct 301985

operational careers and therefore do not neces-sarily match the output indicators from table 8,which are for a 3-year period. Comparable indi-cators exist for the entire 22-year period, 1960-82, in Irvine and Martin’s papers.

Overall, the contribution of Irvine and Mar-tin’s work to research evaluation can be summa-rized as follows:

●

●

●

They have collected, synthesized, and pub-lished a colossal amount of information—all original data— about the scientific per-formance of big and expensive scientific in-stitutes.They have shown that when peers are assess-ing their own fields they can be reliablejudges of scientific performance.Where choices have to be made in a field,among several similar research units compet-ing for resources, Irvine and Martin providepolicymakers with sound information for as-sisting a rational decision.

On the negative side, it is not known how theIrvine and Martin approach would fare in non-Big Science areas. Would the methodology trans-fer, as Irvine and Martin assert, to different cul-tural and research contexts? Even if convergingindicators can validate contribution to scientificprogress or the impact of a research team on itspeer community, judgments of applicability andquality of these findings do not automatically fol-low. These are properties of interpretation, notanalysis. Irvine and Martin tend to confuse the two.

The Irvine and Martin methodology is basedsolely on bibliometric and peer ratings among fa-cilities in the same science, not knowledge-pro-ducing facilities in different sciences. However,the strategic choices between fields are the tough

36

ones in a zero-sum world. Like peer review, “con-verging partial indicators” are useless for strate-gic choices.

Finally, knowledge is produced by scientificcommunities, not individual institutions. There-fore, comparing facilities may bean empty exer-cise. The implication of Irvine and Martin’s rec-ommendations is that reducing or eliminatingfunding to the least cost-effective facility has noadverse effect on progress, and in fact divertsscarce resources to more productive facilities else-where. Such a strategy, however, undermines theknowledge-producing community and runs counterto the view of science as a cultural activity thatintertwines local teams with distant peers throughliterature, informal communication, and the train-ing of new generations of practitioners. These arenot ignored by Irvine and Martin, but they areminimized.

The Center for Research Planning (Coward,Franklin, and Simon) has developed the methodof “bibliometric modeling” based on Institute forScientific Information (ISI) data and co-citationclusters to monitor the research front of a givenspecialty. 18 Each model consists of an intellectualbase and the current work of the specialty. Whenbrought together in a computer, these two setsof papers contain the building blocks of a model:“title words (keywords) of its current papers” and“the demographics of the specialty” (performingorganizations and countries). The “age of its in-tellectual base” is an indicator of the specialty’s“development potential. ”

Working closely with the Economic and SocialResearch Council of the United Kingdom, theCenter for Research Planning (CRP) builtspecialty-specific models and met in workshopswith key participating research teams and tech-nical experts from the respective research coun-cils responsible for the funding. This hands-on ap-proach allowed data to be passed to the scientistsfor their own independent analysis. In otherwords, CRP works with representatives of theknowledge-producing communities being evalu-

‘“H. R. Coward, et al., “ABRC Sc]ence Policy Study: Co-Citation Bibliometric Models” (abridged), presented to the Advi-sory Board for Research Councils, Department of Education andScience, United Kingdom, July 1984, pp. 1-3, 65.

ated and the policy users themselves to increasecredibility and relevance of their studies. As their1984 final report to the Advisory Board for theResearch Councils states:

The models are not intended to function ascomputer-based decision algorithms in the scienceresource allocation process, but should be viewedas a potential decision-support system .19

Though the CRP approach is, like Narin’s,data-intensive and unobtrusive at its source, itis more interactive with the relevant actors. 20 Itis unclear how this iterative and interactive proc-ess affects interpretations. While the modeling no-tion is made explicit by CRP, their methodologyis not as well codified as Irvine and Martin’s. Per-haps recognizing this, CRP is championing inter-active computing with their models to moderatethe suspicions of researchers and policymakersalike. Such interactions will allow users to ask spe-cific questions of a model on-line and receive im-mediate answers. Though this innovation willhave obvious appeal, the jury is still out on itsefficacy.

Other Important Teams

In Holland, the research team of H.F. Moed,W.J.M. Burger, J.G. Frankfort, and A.F.J. VanRaam has carried out extensive comparativestudies of the research productivity of differentdepartments at their home University of Leiden.They have used publication and citation countsto track trends in the quantity and impact of re-search published by individuals and teams in theFaculties of Medicine and Mathematics and Nat-ural Sciences over a 10-year period (1970-80).21

“Ibid.‘“L. Simon, et al., “A Bibliometnc Evaluation of the U. S.-Italv

Cooperative Scientific Research Program, ” Evaluation of L1. S.-JtalvBilateral Science Program (Washington, DC: National Sc]ence Foun-dation, February 1985); J.J. Franklin and H.R. Coward, “PlanningInternational Cooperation in Resource-Intensive Science: Some Ap-plications of Bibliometric Model Data, ” papers presented at the Na-tional Science Foundation symposium entitled “International Co-operation in Big Science, ” February 1985.

*lH.F. Moed, et al., On the Measurement of Research Performa-nce: The Use of Bibliometric Indicators (Leiden, the Netherlands:University of Leiden, Research Policy Unit, Diensten OWZ/ PISA,1983); H.F. Moed, et al., “A Comparative Study of BibliometricPast Performance Analysis and Peer Judgment, ” Saentometncs, vol.8, Nos. 3-4, 1Q85, pp. 149-159; and H.F. Moed, et al., The Appli-cation of Bibliometnc Indicators: Important Field- and Time-Dependent Factors to be Considered, ” vol. 8, Nos. 3-4, 1985, pp.177-203.

37

I n F r a n c e , t h e t e a m o f M i c h e l C a l o n , J e a n - i n t h e r e s e a r c h l i t e r a t u r e . A m a p o f t h e p a i r i n g s

P i e r r e C o u r t i a l , W i l l i a m T u r n e r , a n d G h i s l a i n e of these “co-words” can g ive one a sense o f the

Chartron at the School of Mines in Paris has used structure of a research field .22

the t echnique o f “co -word” ana lys i s to ident i fy

pr inc ipa l problem areas be ing worked on by the

labora tones o f a ma jor French research ins t i tu te ‘A. Rip and M. Courtial, “Co-word Maps of Biotechnologles:and to situate that research in its international An Example of Cogmtive Sc]entometrlcs, ” Sclentometrjcs, VO1. ~

context. Co-word analysis monitors the number 1984, pp. 381-400. M. Callont et al., “_I_he Transltlon Model and

of times that keywords, identified by researchersIts Exploration Through Co-Word Analysls: Using Graphs for Ne-gotiating Research Policles, ” Centre de Sociologle, Ecole des Mines

as describing a research problem, occur in pairs de Pans and Centre Natlonale de la Recherche Sc]entltique, mlmeo.

THE USE OF BIBLIOMETRICS TO EVALUATE RESEARCHAT THE NATIONAL INSTITUTES OF HEALTH

Of all the Federal agencies supporting R&D,NIH conducts the most extensive ex post evalua-tion of its research through bibliometric studiesand other activities carried out at the individualinstitutes. In 1970, the Public Health Service Actwas amended to set aside for evaluation activi-ties up to 1 percent of the funds appropriated toany program authorized by the Act for evalua-tion. Each of the 11 institutes of NIH receives aseparate appropriation from Congress, so thateach can evaluate its own programs. The ProgramEvaluation Branch in the Office of the Directorstudies cross-cutting issues, develops new ap-

proaches to evaluation, and supports the devel-opment of data resources for this purpose .23 Thebudget for evaluation studies at NIH was $5.8 mil-lion in 1985 ($2.8 million from set-aside funds and$3 million from the regular budget). A review ofNIH’s use of bibliometrics illustrate the range ofuseful information that can be produced.

NIH Databases

NIH maintains several extensive databases thatare used for evaluation. The IMPAC databasecontains detailed information about all active re-views and awards of NIH grants, including thenames of all principal investigators, the applicant’sinstitution, the type of grant, the review group,priority score awarded through peer review, thefunding institute, and the amount of support. Twolongitudinal databases developed from IMPACtrack the training or funding history for any in-

Helen Hoter Gee, “Resources tor Research Poitcy Developmentat the National Institutes ot Health, ” typescript, presented betorethe Health Policy Research Working Group, Harvard Unlverstty,Mar. 20, 1985.

vestigator who has applied for NIH support. Aseparate financial database holds year-end dataon all appropriations and obligations since 1950for all NIH institutes and mechanisms of support .24

NIH maintains substantive research classifica-tion systems: Computer Retrieval of Informationon Scientific Projects (CRISP) assigns subdiscipli-nary classification terms to each grant and con-tract, Medical Subject Heading (MeSH) providessubject description and classification informationfor every publication indexed in the Medical Liter-ature Analysis and Retrieval System (MEDLARS)and MEDLINE. MeSH identifies source of research

support, subject, author, title, journal, data, anddescriptors of the research for every indexed re-search article published since 1981. These data-bases are used for literature searches and for eval-uation of NIH activities in a given research area.

NIH uses a special database, MEDLINE, forbibliometric analysis. This database containsrecords of all articles, notes, and reviews that haveappeared since 1970 in a selected group of bio-medical journals, along with the sources of finan-cial support acknowledged by the authors of eacharticle. There are over 300,000 papers in the data-base, as well as a record of nearly 2.5 million ci-tations to those papers. Originally, the 240 jour-nals of the database covered about 80 percent ofthe publications resulting from NIH-supported re-search. The size of the journal base was expandedin 1981 to include the entire MEDLARS systemof the National Library of Medicine, nearly 1,000journals, accounting for 95 percent of NIH-sup-ported research. This extensive database has been

“Ibid.

38

the subject of the bulk of U.S. bibliometric studies,most of which seek to measure the long-term sci-entific payoffs from NIH-supported research.

Bibliometric Studies at NIH

Grace M. Carter of the Rand Graduate Insti-tute conducted the first NIH commissioned bib-liometric study in 1974.25 Carter explored the useof citations as a measure of the research outputof 747 research project grants and 51 programproject grants awarded on a competitive basis infiscal year 1967. The three output measures forresearch grants were the priority score receivedon renewal applications, the production of at leastone frequently cited article, and the average ci-tation rate for publications cited at least twice.Using statistical multivariate analyses, Cartertested a series of hypotheses about the three re-search output measures. Her examination of studysection judgments of renewal applications re-vealed that on average grants proposed for re-newal produced more useful research results thanother grants. She also found that a grant thatproduced a highly cited publication was morelikely to be renewed than one that did not. Thus,peer group evaluation and citation analysis pro-duced comparable results.

In addition, Carter found a high correlation be-tween priority scores on the first grant applica-tion and the number of subsequent publicationsand citations. She also found that research pro-posals perceived by study sections to have a highprobability of being “exceptionally useful” re-ceived higher priority scores and more years offunding than those not so perceived. Carter con-cluded, rather cautiously, that the concept of “sci-entific merit” contains enough objective contentthat different groups of people meeting severalyears apart will agree that one set of grants is morescientifically meritorious than another set ofgrants. 26

‘Grace M. Carter, Peer Review, Gtations, and Biomedical Re-search Policy: NIH Grants to Medical School Faculty, prepared forthe Health Resources Administration and the Office of the Assis-tant Secretary for Planning and Evaluation of the Department otHealth, Education, and Welfare, R-1583-H~ (Washington, DC,HEW, December 1974).

“Ibid., p. v.

Francis Narin furthered the work of Carter byusing bibliometric techniques to obtain quantita-tive indicators of research performance that werein general accord with the intuitive expectationsof the research community .27 He was able to es-tablish a degree of concordance between the struc-ture of biomedical research literature and thestructure of biomedical knowledge, which enabledhim to use bibliometric databases and analysesto demonstrate a number of interesting points:

Utilizing correlational techniques, he wasable to establish correspondence between bib-liometrically measured research productivityindicators and quantitative, nonbibliometricmeasures, including institutional funding andinstitutional ranking based on formal peerassessment.International biomedical publication rates arehighly correlated with the GNP and nationalaffluence (GNP per capita). 28

Changes in U.S. research funding can be as-sociated with changes in the number and con-tent of research publication 3 to 5 years later.Basic biomedical information is both pub-lished and cited by scientists supported bymany bureaus, institutes, and divisions atNIH, forming a pool of fundamental researchknowledge. In contrast, clinical informationis produced and used by a narrower set oflargely clinical researchers. Basic research ismore highly cited than clinical research.Differences exist in the kinds of research pub-lications produced by scientists in medicalschools of different sizes and levels of na-tional prestige. The number of publications

‘-Francis Na.nn, Concordance Between Sub/ect ve and Bibllo-metnc Indicators of the Nature and QuaIity ot Pertormed BlomedlcalResearch, a Program Evaluation Report for the Office ot Program,Planrung and Evaluation, National Institutes 01 Health (Washing-ton, DC: NIH, April 1983); Francis Narin, Evaluative Bib/iomet-rics: The Use of Publication and Citation Analysis in the Evalua-tion of Scientific Activity, monograph prepared for the NationalScience Foundation, Accession #PB2.5z339/AS (Springfield, VA: Na-tional Technical Information Service, March 1976).

‘J. Davidson Frame and Francis Narin, “The International Dis-tribution of Biomedical Publications,” Federation Proceedings, vol.36, No. 6, May 1977, pp. 1790-1795. Frame and Natin investigatedthe U.S. role in international biomedical publication bawd on countsof articles, notes, and reviews in 97s biomedical journals, Thev foundthat the Un]ted States authored 42 percent ot these papers whichwere tar more heavily c]ted than papers trom other countries. Only4 percent of publications were found to orlglnate trom underdevel-oped regions.

.

39

●

●

produced per dollar of research funding is thesame for the large and small institutions, in-dicating neither economies or diseconomiesof scale. However, scientists from the largermedical schools publish their papers in moreprestigious journals, and in a much wider setof subfields than smaller schools. Smallerschools can attain a critical mass of researchactivity only if they concentrate their re-search effort in a select area. In addition,faculty perceptions of the ranking of medi-cal schools are very much in accord with bib-liometric measures of the ranking of the sameschools.The research supported and performed by thedifferent institutes is appropriately concen-trated in the clinical areas corresponding totheir missions.Publications resulting from research con-ducted at NIH are more highly cited thanpublications in the same research areas sup-ported through other sources.

Narin’s work was the basis of the most widely ac-cepted application of bibliometric techniques—the Science Indicators series of the National Sci-

ence Foundation29— and provided the fuel formore extensive use of bibliometrics for evaluationand planning at NIH. Evaluations include the ef-fectiveness of various research support mecha-nisms and training programs, the publicationperformance of the different institutes, the respon-siveness of the research programs to their congres-sional mandate, and the comparative productivityof NIH-sponsored research and similar interna-tional programs.

Bibliometric analysts tested the ability of cita-tion maps and co-citation clusters to detect tran-sitions between basic research and clinical researchin the biomedical sciences. 30 Citation maps of re-search in Lesch-Nyhan syndrome, Tay-Sachs dis-ease, and the effects of the drug methotrexate dis-played the anticipated transitions, though the

‘vScjence zn~cators 1972; 1 9 7 4 : 1 9 7 6 ; 1978; 1980; 1982; 1984,reports to the National Science Board, Nat]onal Science Founda-tion (Washington, DC: U.S. Government Printing Office, ~973,1975, 1977, 1979, 1981, 1983, 1985)

IJ, S. Department of Health and Human Servlc~, App/Ic., tJorIsof Bibliometnc ,\lethods to the Analysis and Tracing ot SclentlficDiscoveries, HHS-MH Evaluation Report NTIS #pB80-210586(Spnngheld, VA: National Technology Information Service, 1981),

extent of the transition varied from case to case.Co-citation cluster analysis identified the basic orclinical orientation of the different research areasbut did not find the sought-for transition points.

Most recently, NIH has undertaken bibliometricstudies to determine the effectiveness of differentresearch support mechanisms. For example, astudy of the research centers’ programs using sub-sequent grant applications and publications as thecriteria, found that a grant to a center is a moreeffective mechanism for supporting clinical re-search than it is for supporting basic science .31 TheDental Institute used bibliometric analysis to de-termine that their centers’ program has been ef-fective in recruiting new scientists to research rele-vant to the institute’s mission. Similarly, theNational Cancer Institute and Francis Narin areconducting a study to identify the contributorsto the most important research findings of the last15 years and to determine where the research wasconducted and the mechanisms of support. Theresearch will test some assumptions of biomedi-cal research grants policy, for example, that theindividual investigator grant is superior to thecontract and that extramural science is better thanintramural .32

Bibliometrics have also been used for evaluatingbiomedical manpower training programs. PeterCoggeshall, a staff member of the National Acad-emy of Sciences (NAS), used the NIH database,NSF grant data, and the NAS Survey of DoctoralRecipients to compare investigators who had re-ceived NIH predoctoral training support with twoother groups—a group that had been trained indepartments that had received training grants anda group that had received no NIH support in anyform. The study concluded that individuals whoreceived NIH predoctoral support produced su-perior subsequent career records in terms of pub-lications and citations, were more likely to beworking on NIH-sponsored activities, and weremore successful in obtaining grants .33

Although Narin’s early work showed that thedistribution of papers supported by each institute

“Gee, op cit, p. 9“LOU Carrese, A s s o c i a t e D\rectc>r tor ProKram Plannlng and

Analvs]s, INatlonal Cancer [nst]tute, pw+unal communication 1°S5“Inst]tute ot Medicine, The Career .~chw~rments ot ,VIH Pre-

doctorai Trainees and Fellows ~Washlngton, DC Xat]onal Acad-emy Press, 1984).

4 0

in the basic and clinical medical disciplines fol-low very closely the institute’s mission, severalinstitutes continue to pursue the use of bibliomet-rics to validate accountability. For example, Narinhas recently used bibliographic methods to evalu-ate trends in pulmonary and hypertension re-search. He found that National Heart, Lung, andBlood Institute actions since the passage of the Na-tional Heart, Blood Vessel, Lung, and Blood Actof 1972 have led to quantifiable progress in theresearch areas listed in the mandate .34 Narin’swork with the National Cancer Institute will beapplied to the same purpose. In addition, the Na-tional Institute of Mental Health is conducting a10-year analysis of the publication record of itsgrantees for purposes of accountability.

In almost all cases, bibliometric studies evalu-ated program performance and conformity toagency or institute mission. In some cases, theyhelped to identify areas for future research fund-ing. The National Institute of Mental Health hasbegun to use bibliometrics and cluster groups toconduct a form of “portfolio analysis. ” Lookingat their program portfolios and the clustering ofresearch publications by field, they are identify-ing leading edges of research that might requiremore support from their institute. 35 Narin hasshown empirically that bibliometric data mayqualify as an important adjunct measure to moresubjective measures applied through peer review.

The Utility of Bibliometrics forResearch Decisionmaking

Bibliometric techniques provide rough indica-tors of the quantity, impact, and significance ofthe output of a group of scientists’ research. Theyare not generally considered valid for measuringthe productivity of individual scientists due todifferences in publishing styles and journal re-quirements, and the questionable validity of small

‘Public Health Service, Bibliographic Methods for the Eva~ua-tion of Trends in Pulmonary and Hypertension Research, NTIS+PB82-159724, 1982.

“Lawrence J. Rhoades, Science Policy Planning and EvaluationBranch, Office of Poiicy Analysls and Coordination, Nat]onai in-stitute of Mental Heaith, penonai commumcation, 1985.

statistical samples. However, Cole and othershave shown that publication counts correlate posi-tively with other measures of individual scientists’research quality such as peer review, Nobel Prizes,and prestige of academic appointment .36

Publication counts give a rough measure of thequantity of work produced by a research teamor facility. Citation counts are an indicator of theinfluence that work has had on the larger scien-tific community. And the number of citations perarticle or the number of highly cited articles pro-vide a rough measure of the significance of thework, since important papers tend to be citedmost often. These indicators can help a fundingagency compare the quantity, quality, and visi-bility of research done by various individuals orinstitutions. They can help identify the strong re-search groups and the relative cost-effectivenessof research sponsored at different centers.

However, they have two important limitationswith respect to research decisionmaking. First,they are entirely retrospective. They have no in-herent future predictive capability, unless one be-lieves that past performance is an indicator oflikely future achievement—not an unreasonableassumption. Therefore, they are more applicableto research program evaluation than to researchplanning. Second, they are not applicable to stra-tegic decisions about resource allocation betweenfields. Most bibliometricians contend that thetechniques can only be validly applied within in-dividual disciplines. Publication and citation prac-tices vary too widely between fields to allow forinterdisciplinary comparisons. This, unfortu-nately, makes bibliometric techniques of limitedvalue for the most important decisions facingagency heads and congressional decisionmakers—allocating resources among fields.

It should be noted that some analysts dissentfrom this view. Derek de Solla Price, in an un-published article for the National Research Coun-cil, argues that the relative strength of different

‘G. A. Coie, The Evaluation ot Basic Research In industrialLaboratories (Cambridge, MA: Abt Associates 1o85 ~ p 43

.

41

fields in the United States can be assessed by com-paring the ratios of the numbers of citations ofU.S. articles in foreign journals to the numbersof citations of foreign articles in U.S. journals byfield, normalized to account for differences in na-tional research “output” by field. Price’s schemeis quite complicated and involves measures ofquality, quantity, and “internationality, ” but itis a first attempt to compare fields using dimen-sionless indicators that have been normalized toremove the effects of different publishing and cit-ing practices between fields. 37

Co-citation analysis enables one to monitorhow specialties or subfields evolve over time. Co-gitation analyses display the relationships amonghighly cited papers by showing how many timessuch papers are cited together in single articles.Based on co-citations, two-dimensional diagramsor maps of specialties can be created which illus-trate the clustering of the most important worksin that specialty, based on the number of citationsBy examining changes in the clusters one can trackthe evolution of the specialty over time. For ex -ample, figures 2A, B, C, and D illustrate the evo-lution of the collagen specialty cluster in biochem-istry between 1970 and 1973. As can be seen bycomparing figures 2D and A, the cluster map hasbecome much larger by 1973, and most signifi-cantly, an entirely new set of research papers hasreplaced the cluster of most important works iden-tified in 1970. This change coincided with the dis-covery of a new substance, pre-collagen, in 1971,which totally reoriented the research front in thespecialty. Thus co-citation maps can, in princi-ple, help one to identify important changes in re-search specialties over time.

‘-Derek de Solla Pr]ce, “Science Indicators ot Quantity andQuality for Fine Turung of United States Investment m Research]n Malor Fields of Science and Technology, ” typesmpt draft, pa-per prepared at the request of the Commission on Human Resources,National Research Council, April 1980, typescript draft.

Even without maps, co-citation cluster analy-sis can help one identify the level of research activ-ity in different specialties. Table 10 takes specialty

Table 10.—Changes in Sample of Continuing Clusters,1970-73

Direction 1970-71 1971-72 1972-73Specialty of change (%) (%) (Ye)Nuclear levels

Adenosine trlphosphatase

Australia antigen

Proton-proton elastics c a t t e r i n g

Ultrastructure ofsecretory cells

Nuclear magnetic resonance

Polysaccharides

Crystallizatlon of polymers

Affinity chromatography

Leukocytes: chonicgranulomatous disease

Collagen

Erythrocyte membranes

Delayed hypersensitivity

58212167

03355

441

5050

0

5012383713504646

8100

00

602020

4013478020

09

64277715

8

4555

0255025542620

72172

4357

055

936443422

10000

670

33

635

3240

06015

560462727

251758672211573013

442531

600

4023542336

757

10000

721414

3353142740335842

3502921,

c - continuing, d = dropping, n - new documents

SOURCE Yehuda Elkana. et al., (eds ) Toward a Merrfc of Science The Adventof ScIerrce Ir?dcators (New York: John Wiley & Sons. 1978). PP 199201

4 2

clusters in biochemistry and shows, for each, thepercentage of key papers that are identical to theprevious year’s, the percentage that have droppedout from the previous year, and the percentagethat are new. As can be seen, the specialties varyappreciably, from crystallization of polymers, forwhich the same papers defined the subfield clusterover all 3 years, to erythrocyte membranes, inwhich 64 percent of the important papers droppedout in 1970, and 80 percent of the papers weretotally new in 1971. If research decisionmakersare eager to fund specialties where new ideas areemerging rapidly, data on the evolution of clus-

ter specialties over time could help to identifyfields of rapid change.38

It should be stressed, however, that most bib-liometncians view publication, citation and co-citation analyses as complements to, not substi-tutes for, informed peer evaluation. All three anal-yses are, of course, ultimately indirect measuresof the scientific community’s peer evaluation ofresearcher’s productivity.

“Eugene Garfield, et al., “Citation Data as Science Indicators,Toward a Metric of Science: The Advent of Science indicators,Yehuda Elkana, et al., (eds. ) (New York: John Wiley & Sons. 1978),pp. 196-201.

Figure 2.–Davelopment of a Speciality Cluster, 1970-73

A. MILLER

PIEZ BiochemrBiochem 1969

1 1

BUTLERBiochem

/

1967

I BORNSTEINBiochem

1966+,

BAILEYBBRC1968

I 1966 IThe figure shows the evolution of the collagen cluster over the 4-year periodof first authors of the highly cited papers and years of publication. Lines11 times in the corresponding source year.

NOTE A IS collagen, 1970, and B IS collagen, 1971

1970-73. Boxes contain the namesconnect papers co-cited at least

43

Figure 2.—Deveiopment of a Specialty Cluster, 1970-73 (continued)

c.

NOsol

1t

44

SCIENCE INDICATORSOne method of assessing the health of the re-

search enterprise is to directly question the scien-tists and administrators involved in it. This is donein a variety of ways in this country. Individualscientists are asked to testify at congressional hear-ings. Federal agencies create scientific advisorypanels to help guide research.

The National Research Council and its constit-uent bodies—the National Academy of Sciences,the National Academy of Engineering, and the In-stitute of Medicine—carry out numerous reviewsof research programs and research fields for theexecutive and legislative branches. The most com-prehensive of these are the Research Briefings andFive Year Outlooks prepared by the Committeeon Science, Engineering, and Public Policy. Sinceall NRC reports are prepared by committees ofscientists, they represent, to some degree, the in-formed, consensus-based peer judgments of thescientific community on the state of the researchenterprise. As a check on the validity of suchreports, the government could support detailedsurveys of the scientific community.

However, scientists’ judgments on the state oftheir own field of research can rarely be totallydisinterested and often reflect the researcher’scharacteristic desire to investigate more problemsin greater depth than available funding will al-low. These inherent biases can be compensated,to some degree, by the use of a variety of scienceindicators that are measured by NSF, NIH, andNRC and published on a regular basis. These in-dicators include the amount of funds devoted bythe Nation to research and development by source,sector, nature of the work, performer, and scien-tific fields; statistics on the distribution of scien-tific and engineering personnel, graduate students,and degree recipients by field, sector, and insti-tution; and the support for graduate educationand training. The NSF Science Indicators tablesinclude funding levels by agency and even pro-gram; the specific institution receiving the funds,

employing the scientists, and training the gradu-ate students; and funding for specific Standard In-dustrial Classification codes within industry. Mostlacking in the policy community is a consensuson which indicators are most relevant and howthe different indicators might be used in combi-nation to measure the health of the research en-terprise. A workshop or report on the use of Sci-ence Indicators to measure the health of theresearch enterprise might be a useful first step inthat direction.

One must remember, however, that all meas-ures or “indicators” of research are inevitablyflawed. Any number describing research is an ab-stract symbol that depicts, imperfectly, only oneaspect of it. Choosing one measure over anotherimplies that the measurement use has made someassumption about what is important. The chosenmeasure has meaning only through interpretation.

Even if an acceptable measure of an aspect ofresearch can be devised, interpretation remainsproblematic:

. . . the inputs [to science]—of dollars, of work-ing scientist, males, females, and Hispanics, grad-uate students, post-docs, and professors—are wellknown and further broken down into industry,government, education, or lost to view. We alsohave counts of outputs —of papers, citations ofpapers, and Nobel Prizes arranged according tonational origin. But how do we know what thenumbers “ought” to be? . . . such indicators helpvery little in determining the health of science inany absolute sense or, more practically, in rela-tion to what it might be if organized and financedat some theoretical optimal level .39

These difficulties illustrate the limitations ofscience indicators for research evaluation.

‘R.S. Morison, “Needs, L=ds, and Indicators, ” Science, Tech.nology, and Human Values, vol. 7, winter 1982, pp. 6-7.

Chapter 4

Research Decisionmaking inIndustry: The Limits to

. Quantitative Methods

.

.

Chapter 4

Research Decisionmaking in Industry:The Limits to Quantitative Methods

This chapter reviews the use of quantitativeanalysis in industry research and development(R&D) decisionmaking at the level of the individ-ual firm. OTA reviewed articles, surveys, andreports, and interviewed several research man-agers. Very little systematic information is avail-able about industry’s use of quantitative modelsin research resource allocation. A few surveyscover limited numbers of firms and are not nec-essarily representative. Much of the material isin the form of anecdotal accounts of an individ-ual firm’s uses of certain methods, providing noinformation on whether other firms have adoptedsimilar approaches. Relatively more informationis available about noneconomic quantitative meth-ods for project selection than about economicmodels. The literature, as well as OTA interviews,demonstrates the limited practical utility of quan-titative techniques for research decisionmaking in

industry, and the reliance on subjective judgmentand good communication between R&D, manage-ment and marketing staffs in the decisionmakingprocess.

In the private sector, R&D is an investment thatmust compete for corporate support with otherinvestment opportunities such as plant expansionor new product marketing. Program and labora-tory directors must defend the value of their re-search to top management and decide what mixof projects is best for the firm. Project managersmust determine whether their projects are pro-ceeding as planned and whether expected payoffswill justify costs. This chapter looks in turn at theuse of quantitative methods in review and evalu-ation of ongoing research, new research projectselection, and research resource allocation as partof strategic planning.

REVIEW AND EVALUATION OF ONGOING RESEARCH ACTIVITIES

As part of their normal operations and man-agement, firms periodically review research pro-grams, projects, and staff to assess progress anddetermine the contribution that individual re-searchers and research groups are making to thefirm’s goals. These reviews can justify researchexpenditures to management, assist in budget and.program planning, or evaluate personnel per-formance.

Few firms use quantitative methods to reviewongoing research. In 1982, Schainblatt surveyed34 R&D-intensive firms about methods of meas-uring research productivity.1 The survey focusedon the groups, programs, or other organizationalunits, rather than on individual scientists or engi-neers. Managers in only four firms reported using

performance or output measures as part of theirprogram reviews. Only 20 percent of the firmsroutinely collected any kind of productivity data,and 20 of the 34 firms reported using no produc-tivity-related measures at all.

Several respondents said they had tried foryears to measure R&D productivity but had notbeen successful. Noted one research manager,“We . . . came to the conclusion that there is nogood way to do it on a week-to-week or month-to-month basis. ” Managers doubted that R&Dproductivity measures were meaningful. Accord-ing to one manager, “Attempts to quantify bene-fits of R&D have led to monstrosities that causedmore harm than good. ”2

A. Schalnblatt, “How Companies \leasure the Productlvlty ofEng]neers and Sc]entlsts, ” Research Management, May 1982, pp.10-18.

‘Gerald A.Laboratories

Cole, The Eva]uatlon oi Basic Research [n lndustndi(Cambridge, MA: Abt Associates, 1985), p. 59.

47

48

Macroeconomic Models

The retum-on-investment (ROI) model is oneof a family of economic analysis tools, along withdiscounted cash flow analysis and present valueanalysis, which business managers use as aids toinvestment decisions. These methods are com-monly applied to decisions where uncertainty islow. As we have seen in chapter 2, research in-vestment decisions entail considerable uncer-tainty. Moreover, the ROI methods carry a biasagainst the long-term, high-risk, and high-uncertainty projects like basic research, and workin favor of high-vield, short-term investments.3

The bias increases in periods of high inflation likethe 1970s.

Despite the fact that economists have typicallyviewed R&D activity as an investment, a num-ber of scholars recently have criticized the mis-use by private sector managers of quantitative fi-nancial techniques for evaluating investments inR&D and technology more generally. Hayes andAbernathy argue that the application to invest-ment decisionmaking of principles of portfoliomanagement has led U.S. firms to underinvest innew technologies:

Originally applied to help balance the overallrisk and return of stock and bond portfolios, theseprinciples have been applied increasingly to thecreation and management of corporate portfolios—that is, a cluster of companies and product linesassembled through various modes of diversificationunder a single corporate umbrella. When appliedby a remote group of experts primarily concernedwith finance and control and lacking hands-on ex-perience, the analytic formulas of portfolio the-ory push managers even further toward an ex-treme of caution in allocating resources. 4

Similarly, Hayes and Garvin have argued thatpresent-value analysis of R&D investment deci-sions has led to a systematic bias against such in-

‘J.E. Hodder and H.E. Riggs, “Pitfalls in Evaluating Risky Proj-ects, ” Hanrard Business Review, January-February 1985, pp. 128-13.s; and G.F. Mechlin and D. f3erg, “Evaluating Research-ROI1s Not Enough,” FLwvaru’ Business Review, September-October 1980,pp. 93-99.

‘Robert H, Hayes and William J. Abernathy, “Managing ourWay to Economic Decllne, ” Harvard Business Review, vol. .s8, No.4, 1980, pp. 67-77, reprinted in Surwval Strategies for Americanindustry, Alan M. Kantrow (cd. ) (New York: John Wiley & Sons,1983), pp. 15-3S (quotation IS from pp. 22-23, 1983 reprint).

vestments, due in part to the use of discount or“hurdle” rates for such decisions that are too highsRather than relying solely on quantitative tech-niques for the evaluation of R&D investments,these authors argue that managers must developan understanding of the underlying technologies,and apply informed judgment in making such de-cisions.

ROI methods, as applied to R&D projects, esti-mate the economic value of research and compareit with the cost to the organization. For example,a firm could estimate the sales or revenues gen-erated or expected from a new product resultingfrom research efforts. Alternatively, the firmcould estimate the share of total profits or sav-ings attributable to research. The financial resultsare usually discounted to reflect the time valueof money.

One major problem with such methods is thatit is difficult to apportion profits generated by aproduct developed in the past among research,development, and marketing activities, all ofwhich contribute to the process. Even if one couldattach accurate figures to the present payoff frompast research, such calculations offer little guid-ance for current decisions, which are occurringunder different technological, management, eco-nomic, and organizational conditions. b In addi-tion, companies often acquire R&D through pur-chase, corporate acquisition, or merger; or theymay sell R&D themselves.

ROI techniques seem most applicable to justify-ing the value of ongoing research to top manage-ment by placing a value on past research. Such

‘.+s these techniques have gained ever wider use In Investmentdeclslonmakmg, the growth ot capital Investment and R&D spend-ing m this country has decllned We summ]t that the dscounttngapproach has contributed to a decreased willingness to tnvest for thefollowing reasons: (1) It IS otten based on mwperceptlons of the pastand present economic environment; and (2) It IS biased against in-vestment because of critical errors In the way this theory IS applledBluntly stated, the willingness of managers to wew the tuture throughthe reversed telescope of discounted cash flow IS shortchanging thetuture of the:r compames.

Robert H. Hayes and David A. Gamin, “Managing As If Tomor-row Mattered, ” Harvard Business Review, VOI. 00, No. 3, pp. 70-79, reprinted in Survival Strategies for American Industry, AlanM. Kantrow (cd. ) (New York: John Wiley & Sons, 1983), pp. 30-51, (quotation is from p. 37, 1983 reprint).

“B. Twiss, ,Mmagmg Technological lnrro~’~tlc?n ( SW ) (Irk Long-man, 1980 ), yp. 121-122; and D W. Collier, ‘%leasurlng the Per-formance of R&D Departments, ” Research ,Jfanagement, VOI, 20,No. ~, 1977, pp. 30-34.

49

techniques do not appear very helpful for deter-mining changes in a research program’s budget,or deciding the fate of particular project. The ROImethods and other cost/benefit methods, as dem-onstrated below, are more applicable to decisionsinvolving setting priorities among a set of appliedresearch projects.

Business Opportunity Techniques

Collier describes a “business opportunity” tech-nique that avoids some of the pitfalls of ROI ap-preaches. ’ It is based on the notion that the pri-mary objective of industrial research is to identifyand define business opportunities that can be ex-ploited commercially. Management evaluates theperformance of research staff by comparing itstechnical accomplishments against a set of previ-ously established objectives—for example, to de-velop a new control system with certain perform-ance characteristics. Next, when the project iscompleted and is ready to be transferred to theproduction and marketing departments, total ex-pected sales revenues are estimated and discountedto the present. The result, when divided by theproject’s cost, is a return on research figure which,if summed across all completed projects, yieldsan estimate of the research department’s value tothe company for that year.

The business opportunity method depends onthe accuracy of future sales estimates and of prod-

uct development time, which are subject to sub-stantial error. Thus, uncertainties in the ROImethod’s allocation of credit for profits retrospec-tively among the various factors are replaced bythe business opportunity method’s uncertaintiesabout future payoffs. Collier evidently assumesthat there is a higher level of certainty about fu-ture profits than about the contribution to prof-its of past activities. Nevertheless, Collier statesexplicitly that neither the ROI nor the businessopportunity method should be used to evaluatebasic research, ’ probably because particular basicresearch projects and programs are difficult to tieuniquely to economic impacts, which may wellbe separated from the research by many years andinstitutional boundaries.

Little is known about the use of other formalevaluation techniques such as bibliometrics, pat-ent counts, and colleague surveys for peer review.DuPont, Bell Laboratories, and other large indus-trial research establishments carry out intensive,annual reviews of their scientists’ work. These re-views stress the scientists’ contribution to scienceand technology, particularly in areas of strategicinterest to the company. The reviews are per-formed by peers in the fields of research con-cerned. 9 No evidence suggests that this practiceis widespread. One complicating factor is thatmuch industrial R&D is shared work, so attribut-ing some portion of its output to one individualis difficult.

‘Ibid.‘Cole, op cit., p. 38.

R&D) PROJECT SELECTION

Relatively more information exists about indus-try’s use of formal, quantitative techniques to se-lect R&D projects and shape research portfolios.An extensive literature on models and methodsexists and it includes some surveys on their use.These algorithms or heuristic devices help man-agers assign values to projects, groups of projects,or other investments. The approaches fall intofour categories:

‘Ibid., p. 59

3. constrained optimization or portfolio mod-els, and

4. risk analysis or decision analysis models.

Scoring Models. When scoring models are used,each project is rated against a series of relevantdecision criteria. Scores for each project are com-bined through addition or multiplication to de-velop a single project score.

1. scoring models,2. economic models,

In a typical application, all candidate projectsare scored and ranked from highest to lowest.

50

Since costs are associated with each project, theallocation decision involves simply going downthe list of projects until the available funds havebeen exhausted. This procedure does not considerthe effect of variations in project budgets, mar-ginal returns from varying project funding levels,interactions among projects, or changes in annualbudget levels over the life of the projects.

Scoring models have the least demanding in-put data requirements of the four categories ofmodels. They are designed to incorporate noneco-nomic criteria, and can operate on input data inthe form of subjective estimates from knowledge-able people. The assumptions underlying scoringmodels are relatively undemanding: only that ex-plicit evaluation criteria and a way to quantifyeach evaluation be developed. While expert judgesmay be used as the source of qualitative inputdata, eventually these data must be expressedquantitatively to be used in a scoring model. Thechoice of algorithm to convert qualitative data toquantitative scores is arbitrary and subjective.

Economic Models. With economic models,projects are rated against a series of economic cri-teria such as expected rate of return. A single fig-ure of merit is produced, typically reflecting theratio of the present value of earnings from theproject, including the probability of project suc-cess, to discounted money flow or investment.These are essentially capital budgeting models.Economic models accept only quantitative databased on estimates of the financial performanceof the project over a specified planning horizon.These estimates are often generated by programor project managers or by panels of experts. Theypossess no greater intrinsic validity than data de-veloped for scoring models. This is particularlytrue when uncertainties about the technical andmarket performances of the technology are high,a situation frequently encountered in early stagesof applied research. Any estimate of future projectbenefits requires subjective input from some well-informed respondent or group of respondents. 10

N’. R. Baker, “R&D Pro]ect SelectIon Models: A Assessment, ”IEEE Transactions on Engineering Management, EM-21, Novem-ber 1974.

Like scoring models, economic models producea single figure of merit that is independent of thefigures of merit for competing projects. The sim-plest application is to rank projects and fund thosescoring highest until the available funds are ex-hausted. The narrow focus of economic modelsmay limit their usefulness for government re-search, where cost and economic gain are onlypart of a much larger set of criteria.

Constrained Optimization or Portfolio Models.These models measure a program’s potential tomeet a goal, usually a series of economic objec-tives, subject to specified resource constraints. Un-like scoring and economic models, the focus is ona mix of projects rather than a simple ranking ofindividual projects. In portfolio analysis, mathe-matical programming techniques are used to eval-uate the allocation of resources among candidateprojects. The method requires an understandingof the relationship among resource inputs, tech-nological performance, and marketplace response.The decisionmakers must agree that funding justthe right mix of resources among projects is thekey to effective R&D management.

While demanding better data quality and un-derstanding of underlying technical and economicprocesses, portfolio models can handle multipleconstraints and different budget levels over differ-ent years in the planning horizon. But the use ofthese models implies a level of management con-trol and flexibility to reallocate resources that maynot exist in many situations, particularly in gov-ernment. 11

Risk Analysis or Decision Analysis. Thesemodels produce an expression for the expectedutility of each set of alternative budget allocationsamong a set of research projects. Models usingdecision analysis have the most complex data re-quirements, since inputs must be in the form ofprobability distributions. As in the case of port-folio analysis, considerable understanding of under-lying processes must exist if the benefits of deci-sion analysis are to be realized. Decision analysisincorporates expert judgments as well as “objec-tive” data.

‘K. G. Feller, A Rewew ot .Jfethods tor EL’J/uJt)ng R&D ( L)\’er-more, CA: Lawrence Llvermore Laboratory, IWO), p. 19

51

Use of R&D Project Selection Modelsin Industry

A 1964 survey of the use of quantitative R&Dproject selection methods concluded that whilenumerous models and techniques had been pro-posed, available data showed “little thorough test-ing and only scattered use of the proposed meth-ods. ”12 A second survey published 2 years laterled to a similar conclusion:

The practice of project selection in industry andgovernment is dominated by . . . methods de-pending heavily upon individual or group judg-ment and using very little quantitative analysis.The use of cost and return estimates is common,but very few organizations employ any formalmathematical model for combining these esti-mates and generating optimal project portfolios. 13

Rubenstein pointed out in 1966 that the use ofquantitative methods by R&D organizations hadnot increased appreciably since 1950, and that thereasons for this had more to do with the natureof the available data and the R&D decision proc-ess than with the sophistication of availablemodels.

In a 1968 study of the R&D project selectionpractices of 36 firms, Dean found that formal,quantitative models were not widely used. Sim-ple scoring models employing only a few criteriasuch as probability of technical success, estimatedtime to completion, cost, and size of net marketgain were the only mathematical models that hadbeen tried.14 Meadows’ 1968 report on practicesin the R&D labs of five major companies foundthat the margin of error in estimates underminedthe usefulness of the methodologies. 15 He providesa telling example:

‘N. R. Baker and W.H. Pound, “R&D Prolect Selection: WhereWe Stand, ” IEEE Transactions on Engineering Management, EM-21, November 1974, p. 130.

‘3A.H. Rubenstein, “Economic Evaluation of Research and De-velopment: A Brief Sumey of Theory and Practice, ” The journalof Industrial Engineen”ng, vol. 17, November 1966, p. 616.

“B. V. Dean, “Evacuating, Selecting and Controlling R&D Proj-ects, ” Research Study 89 (New York: American Management Asso-ciation, 1968).

“D. L. Meadows, “Estimate Accuracy and Project SelectIonIModels in Industrla[ Research, ” industrial .%lanagement Rewew,spring 1968, pp. 105-119

If estimates with only a ten percent error wereinserted in the formula, they could conceivablylead management to calculate a higher profit ra-tio for a project actually expected to lose moneyfor the firm than for one expected to return 230percent on the money invested in its development.This sensitivity of the model’s output to error isimportant in view of the fact that no laboratoryyet studied has had estimates (of error) averag-ing as little as ten percent. 16

Mansfield, who has conducted numerous stud-ies of R&D and innovation in industry, noted ina more recent article that:

. . . most companies . . . have found it Worth-while to make economic evaluations of projectproposals and continuing projects, often adapt-ing such capital budgeting techniques as rate ofreturn or discounted cash flow to the task athand. 17

But he goes on to say that the nature of the tech-niques used will vary, depending on the stage ofresearch. Early on, when costs are low and un-certainty high, project screening will be quick andinformal. Later, the larger labs make some use ofquantitative methods:

In some labs, they [quantitative methods] aretaken quite seriously indeed; in others they arelittle more than window dressing for professionalhunches and intra-company politics . . . Themore sophisticated types of models have not beenextensively used. 18

Although it is important to distinguish amongthe various research activities when discussing theuse of such models, surveys rarely do so. Onenotable exception is a 1971 study of project selec-tion practices by a task force of the Industrial Re-search Institute. The task force studied 27 com-

panies and classified their R&D programs intothree types: exploratory, high risk business de-velopment, and support of existing business, 19

Among the firms studied, those engaging in ex-ploratory R&D generally used simple, unsophisti-

‘61 bid., p. 116.‘7 Edwin Mansfield, “How Economists See R&D “ Research

Management, vol. 25, 1982, p. 25“Ibid.‘“R, E. Gee, “ A Sur\’ev ot Current Project Se)ecf)on Pract]ce5

Research ~Llanagernent, vol. 14, September 1~~1 pp 3S-45

52

cated selection procedures. Decisions on fundingwere based on a page or two of qualitative infor-mation or a simple rating scheme. Decisions onhigh risk business development projects wereoccasionally supported by more sophisticated,quantitative techniques such as standard economicprojections. There was very limited use of quan-titative methods for dealing with uncertainty. Fordecisions about projects in support of existingbusiness, on which quantitative data with verylow uncertainty could be brought to bear, stand-ard economic projections were widely used. A1985 survey indicates little has changed: indus-try managers rely on qualitative evaluation ofbasic research programs and proposals.

There is substantial agreement that the less com-plex scoring models with less demanding input re-quirements are more appropriate for earlier stagesin the R&D process than the more analyticallysophisticated economic and linear programmingmodels. One literature review concluded that eco-nomic models are too quantitative for evaluatingeven applied research efforts; they can help onlyin identifying the information needed to make aqualitative estimate of the economic merit of ap-plied research projects.

20 Twiss, in the second edi-tion of his respected text on the management oftechnological innovation, finds little value in so-phisticated analytic models for R&D projectselection:

While the formulae may give satisfactory sym-bolic representation, it is doubtful whether theyprovide a mechanism of much operational value.The judgments involved are so complex there isa great danger of the formulae being used to ap-ply a veneer of pseudo-quantification to supportdecisions which have already been taken on differ-ent considerations . . . If the data is poor they arelittle better than descriptive representations of theproblem. When applied to estimates of the orderof inaccuracy discussed earlier, they can do a posi-tive disservice by concealing in a simple index themagnitude of uncertainties. However, there aresome types of R&D work where it is possible toassess both the benefits and costs to a high de-

●

‘WM. Burnett and D.J. Monetta, Applied Research Pro!ectSelectIon [n ,%lission-Oriented Agenaes: An Approach fWtishmg-ton, DC: U.S. Department of Energy, Assistant Secretary tor EnergyTechnology, Division of Power Systems, 1978).

gree of accuracy. These are usually developmentprojects.21

The basis for judging the appropriateness ofdifferent types of project selection models fordifferent stages of R&D should not rest on dis-tinctions between quantitative and qualitativedata, since it is always possible to assign a num-ber to qualitative data using some arbitrary al-gorithm. Rather, the association should be basedon the level of certainty involved in the estimatesof the probability that technical and market per-formance goals will be achieved at a certain costwithin a specified time. In basic research and inthe early stages of applied research, these factorscan be predicted with little certainty. (See box B.)

Reasons for Levels and Patternsof Observed Use.

Industry managers recognize that attempts tolink basic research activities directly and quan-titatively to any kind of “payoff’’—new products,profits, corporate image internal consulting,scientific knowledge, or personnel recruitment—are flawed and of limited value. The uncertain-ties are too great, the causal paths too diffuse, thebenefits too difficult to measure, and the time-frame too extended. Basic research usually rep-resents a small fraction (5 to 10 percent) of R&Dbudgets, and is not subject to the same financialscrutiny as applied research and developmentactivities.

The realities of the research process account forthe limited use of quantitative project selectionmodels. The increasing sophistication of themodels has not improved their acceptance. In fact,the increased sophistication may create as manylimitations as it removes. z’ Fundamental inade-quacies in the data required greatly limit theirvalue. Studies of company estimates of projectcost and time requirements show that they areusually highly inaccurate. Mansfield, et al. ,23

found that in one drug firm, the average ratio ofactual to estimated development costs exceeded

‘lTwIss, op. cit., p. 1 3 5 .‘E. P. Winkofsky, et al., “R&D Budgeting and Prolect 5electton.

A Review of Practices and Ivlodels. ‘ T1.\fS Studies )n the .\ fJnwe-ment .%ences, VOI. 15, 1980, p. 1Q2

“Mansfield, op. clt

53

2

Box B.—Differences Between Implicit Assumptions of Project Selection Modelsand Typical Decision Environments

Implicit Assumptions Typical Decision Environment1.

2.

3.4.

5.

6.

7.

8.

9.

10.

A single decisionmaker in a well-behavedenvironmentPerfect information about candidateprojects and their characteristics; outputs,values, and risks of candidates known andquantifiable.Well-known, invariant goals.Decisionmaking information isconcentrated in the hands of thedecisionmaker who has all the informationneeded to make a decision.

The decisionmaker is able to articulate allconsequences.

Candidate projects are viewed asindependent entities to be evaluated ontheir own merits.A single objective, usually expected valuemaximization or profit maximization, isassumed and the constraints are primarilybudgetary in nature.The best portfolio of projects isdetermined on economic grounds.The budget is “optimized” in a singledecision.A single, economically “best” overalldecision is sought.

2.

3.4.

5.

6.

7.

8.

9.

10.

Many decisionmakers and many decisioninfluencers in a dynamic organization.Imperfect information about candidateprojects and their characteristics; projectoutputs and values are difficult to specify;uncertainty accompanies all estimates.Ever-changing, fuzzy goals.Decisionmaking information is highlysplintered and scattered piecemealthroughout the organization, with no onepart of the organization having all theinformation needed for decisionmaking.The decisionmaker is often unable orunwilling ,to state outcomes andconsequences.Candidate projects are often technicallyand economically interdependent.

There are sometimes conflicting multipleobjectives and multiple constraints, andthese are often noneconomic in nature.

Satisfactory portfolios may possess manynoneconomic characteristics.An iterative recycling budgetdetermination process is used.What seems to be the “best” decision forthe total organization may not be seen asbest by each department or party, so thatmany conflicts may arise.

SOURCE: Maptd from William E. Souckr, “A System for brig R&D Prqeci Evaluation Mockh,” Rcscamh tdarugrm alt, September 197D, pp. 29-37.

to 1 development time required exceeded esti- to the potential unreliability of quantitative evalu-ates by a factor of almost 3. A more recent study ation of basic research activities.of major innovations developed during a 5-year

The reasons for lack of reliance on models forperiod by a large U.S. company indicated that ini- . .tial estimates of an R&D project’s expected prof- research decisionmaking include:

itability were no more reliable than the drug firm’scost and time estimates. “The chances were about ●

50-50 that the estimated discounted profit froma new product or process would be more than

●

double or less than half the actual discountedprofit .“24 The inaccuracies of such measures point

●

“Mansfield, op. cit., p. 26

inadequate treatment of multiple, often in-terrelated criteria;inadequate treatment of project interrelation-ships;lack of explicit recognition and incorporationof the experience and knowledge 01 the re-searchers and managers;

54

● inability to recognize and treat nonmonetary Mansfield adds that many models fail to recog-aspects of research programs that are diffi- nize that R&D is a process of uncertainty reduc-cult to understand and use; and tion—in effect, buying information.26 Thus, tech-

Ž inadequate treatment of program and staff nical failures are successes in that they provideevolution. 25 valuable information.

~Winkofsky, et al., op. cit., pp. 191-192. “Mansfield, op. cit., p. 25.

STRATEGIC PLANNING AND RESOURCE ALLOCATIONThroughout the 1960s, company managers gen-

erously funded “open-ended” research—researchthat was not necessarily directed toward increas-ing corporate profits. However, corporate invest-ment in R&D decreased significantly in the 1970s.Dr. Alan Frohman, a management consultant andfaculty member at Boston University’s business

. school, believes management’s attitude towardsinvesting in R&D:

. . . has seesawed from unquestioned support andoptimism in the 1960s to withdrawn support anddiscouragement . . . in the 1970s.“ In the 1960s, the attitudes were evidenced bythe building of large, well staffed laboratories,often remote from the businesses. During the1970s, the major, painful cutback in both expend-itures and staffing documented management’s dis-couragement with the contribution of technologyto the bottom line.27

Giorgio Petroni, a professor who has studied thehistory of management, contends that this “dis-couragement” was caused by the lack of atten-tion given to strategic planning.

During the 1970s, management, even in “tech-nology intensive” enterprises, showed little under-standing of the need to develop technologicalexpertise within their organizations. Top manage-ment often did not understand the full importanceof technology as an element of competitive strat-egy. 28

Petroni and Frohman are not the only scholarsto cite the failure of management to plan for tech-nological innovation as a major factor in the re-

‘ -Alan L. Frohman, “Managing the Company’s Technologica lAssets, ” Research Management, September 1980, pp. 20-24.

%iorgio Petroru, “Who Should Plan Technological 1nnova-tlon?” Long Range Planning, vol. 18, No. 5, 1985, pp. 108-115.

duction of profits in American industries. In fact,in 1980, when Frohman’s study was published,several major literature reviews emphasized whatAlan Kantrow has called “the strategy-technologyconnection. “29

After reviewing the research literature from the1970s, Kantrow concluded that there is no ra-tional justification for separating technology fromstrategy:

Technological decisions are of fundamental im-portance to business and, therefore, must be madein the fullest context of each company’s strategicthinking. This is plain common sense. It is alsothe overwhelming message of this past decade’sresearch .30

Based on the little that was known, Kantrow ten-tatively identified the key elements of corporatetechnology strategy:

. . . good communications, purposeful allocationof resources, top-level support within the orga-nization, and careful matching of technology withthe market .31

While scholars and managers alike knew littleabout technological planning in 1980, they knewa good deal about the theories and practices asso-ciated with strategic management. In the same is-sue of the Harvard Business Review that featuredKantrow’s article, a review by Frederic W. Gluck,et al., stated: “for the better part of this decade,strategy has been a business buzzword.”32 The in-

“Alan M. Kantrow, “Keeping Informed: The Strategy-Technol-ogy Connection, ” Harvard Business Review, July-August 1980, pp.6-21.

‘ibid., p. 6.“lbId, p. 11.‘ : Frederic W. Gluck, et al., ‘Strateg]c Management tor Com-

petitive Advantage, ” Harvard Business Review, July -~ugust 1980,p. 154.

55

crease in the use, and misuse, of this term is im-portant because it signifies a shift in managers’planning approach from technicalities “to substan-tive issues affecting the long-term well-being oftheir enterprise. “33 Despite this change, mostmanagers did not apply these long-term strategiesto their R&D divisions in the 1970s.

However, it is important to note that duringthis period there was a significant countertrend.34

The verbose title of a 1973 Chemical Week, tellsthe story: “Research Gets the Word: If It Doesn’tFit, Forget It— It’s The New Way of Life: R&DMust Mesh Closely With Corporate Goals. ”35 In1976, Union Carbide’s R&D director told a re-porter: “R&D is too important to be left to theR& D’ers; R&D is the future analog of today’s cap-ital expenditures. ”36 The vice president of Celaneseagreed: “At the high cost of R&D, we can nolonger afford to plan and manage it in a randommanner. It has to be very closely tied to strategicbusiness planning. ”37

Eastman Kodak, a pioneer in industrial re-search, was one of the first companies to makea concerted effort to incorporate R&D issues intoits strategic planning process in the 1970s. Thisrepresented a major change in the company’sR&D policy from the 1960s when research direc-tor C.E. K. Mees articulated Kodak’s hands-offpolicy:

The best person to decide what research workshall be done is the man who is doing the research,and the next best person is the head of the depart-ment, who knows all about the subject and thework; after that you leave the field of the best peo-ple and start on increasingly worse groups, thefirst of these being the research director, who isprobably wrong more than half of the time; thena committee, which is wrong most of the time;

‘] Ibid, p. 154.“Kantrow, op. c]t., does believe that managers’ awareness “of

the need to Incorporate technological issues within strategic deci-sion making” (p. 6) grew during the 1970s. However most of hisarticle emphasized management’s refusal to see the connection be-tween technology and strategy, Kantrow (personal communication,1985) said that he believed corporate managers’ views of planningtechnological innovation have evolved over the past three decades(as opposed to swinging from one extreme to another).

‘: Edward D We]] and Robert R, Cangeml, “Llnklng Long-RangeResearch to Strategic Planning, ” Research )bl.anagement, Llav-june

1983, p 33‘Ibid‘Ibtd.

and finally, a committee of vice-presidents, whichis wrong all the time.38

In 1978, Kodak formalized the relationship be-tween corporate management and R&D by estab-lishing a technological affairs committee .39 FormerKodak Vice President W.T. Hanson, Jr., outlinedthe five tasks given the committee:

1.

2.

3.

4.

5.

To assess long-term technical opportunities asthey emerge from the basic research envi-ronment.To establish broad goals for the commitmentof Kodak R&D resources. These goals shouldmeet short-term product and process needs aswell as long-term technology needs.To ensure that resources are properly allocatedto develop the technology necessary to sup-port the longer range business objectives.To approve major corporate product pro-grams including specific goals which encom-pass the following:—schedule,—specifications of features and functions,—resources required,—corporate return, and—assessment of risk.To monitor progress in corporate projects andapprove any changes which have an impacton corporate goals.

Corporate projects as well as the committee’sprogress were monitored in weekly meetings,which were chaired by the Chief Executive Offi-cer (CEO). The meetings included top manage-ment and the staff directly responsible for projectsunder review. To Hanson, these sessions repre-sent top management’s “real commitment” toR&D planning, an idea that was almost unheardof during Mees’ tenure.

Kodak’s decision to establish a technical advi-sory committee was not unique. DuPont andMonsanto, two major chemical manufacturersthat are now entering the field of life sciences, in-creased the power of already existing committeesin order to streamline the R&D budget allocationprocess. Prior to 1979, DuPont had establishedan executive committee to oversee the activitiesof the R&D divisions. According to Robert C.Fortney, Executive Vice President of R&D, “the

“Ibid.‘“W. T. Hanson, Ir “Plannlng R&D at Eastman Kodak Re-

search Management, July 1978, p 24

56

head of central research and each of the individ-ual operating departments had a liaison arrange-ment with different members of the executivecommittee who loosely kept track of the wholething.” In 1979, the company was reorganized,and the loose arrangement between the variousdivisions and top management was replaced byone that was more structured. Fortney believesthis change led to “greater corporate involvementin deciding how much of [the total budget] wouldbe in one field, and how much of it would be inanother field. ”40 This increase in corporateinvolvement has forced the members of the ex-ecutive committee to become more informed. Forexample, Fortney meets bimonthly with the 11 re-search directors in the company, reads quarterlyresearch progress reports from the operating de-partments and the engineering department, re-views monthly reports from the central researchand development department, and listens to pres-entations from the first line people on the resultsof their research, usually two or three times amonth. In addition, he talks informally with theresearch directors about their budget plans .41Through these formal and informal meetings,which enable corporate management and R&Dmanagers to share information, DuPont has triedto include technological issues in its corporatestrategy.

Like Fortney, Howard Schneiderman of Mon-santo strengthened an underutilized committeestructure when he became the Senior Vice Presi-dent for R&D in 1979. Three committees are nowinvolved in the budget allocation process. Thetechnology advisory council, chaired by Schnei-der-man, is composed of the directors and generalmanagers of R&D and technology. Most of thecouncil meetings focus on the administrative con-cerns that effect the management of the scientificenterprise. The council is also a forum in whichproposals for new or continuing research pro-grams are suggested and discussed. Then, the tech-nology review committee, which is also chairedby Schneiderman and which includes the com-pany’s senior executives, evaluates these programsby determining whether or not they are commen-

surate with the corporation’s goals. Finally, theprograms’ budgets are evaluated and then ap-proved or rejected by the executive managementcommittee, which is composed of top manage-ment (including Schneiderman) and which ischaired by the company’s CEO. Schneidermancontends that the change in Monsanto’s view ofthe importance of R&D (which is reflected in thechange in the budget allocation process) cameabout as a result of the company’s decision to shiftits emphasis from producing industrial chemicalsto the field of life sciences:

One way to put it is that Monsanto is now intomore brain-intensive and less raw material andcapital-intensive businesses than we have been be-fore. And that has some enormous consequencesfor the way the corporation thinks about research.That is why . . . R&D suddenly moves forwardin the corporation’s thinking.

In the case of, say commodity chemicals, poly-styrene, you don’t spend an enormous percent-age of your sales on research. You have to spenda reasonable amount, but not 5 or 6 percent. Cer-tainly not 10 percent! An awful lot of money goesinto building a plant, and you’re spending a loton your cement in the ground. So the big deci-sions are capital decisions.

Now the really big decisions are R&D decisions.You’re going to see this not in Monsanto alone,but in other companies too. ’2

The use of committees and other formal meansof communication at Monsanto, DuPont, and Ko-dak is not at all coincidental. What Schneiderman,Fortney, Hanson and others have learned is thatR&D budgeting is “an information and commu-nication process. ” While it is true that each com-pany allocates its resources differently, severalgeneralizations about the budget process can bemade. After reviewing the literature from the1970s, Winkofsky, et al., formulated and substan-tiated these observations:

● the processes are multiperson, involvingmany persons throughout the organizationalhierarchy;

● the processes are multilevel, involving or-ganizational entities at different hierarchies;

%fichael F. Wolff, “An Interwew With Robert C. Fortney, ” Re-search Management, January/ February 1984, p. 16.

‘] Ibid., p. 17.

‘ -D a v i d W e b b e r , “Chlet Scient is t Schnetderman hlonsanto sLove Affair With R& D,” Chem~cai & Eng/neer]ng iN’ews, Dec. 24,1984, p. 11.

57

the processes are iterative;proposals are passed upward through the hi-erarchy;resource allocations are passed downwardthrough the hierarchy;goals in implicit or explicit forms are passeddownward through the hierarchy;the processes are multicriterion in nature;there is no unique set of criteria used by allfirms;areas of concern which are considered bymany firms include: R&D costs and the prob-ability of technical success, manufacturingcosts and the probability of commercial suc-cess, market potential and the probability ofmarket success, and contribution to cor-porate goals;different levels of the hierarchy may usedifferent evaluative criteria; andthe R&D budgeting process may be periodic,continuous, or periodic-continuous. 43

This list contains no mention of the use of finan-cial models. Winkofsky, et al., believe that thecomplexity of the allocation process “often con-founds the formulation of mathematical models. ”Because different levels of the hierarchy may usedifferent evaluative criteria, there is no firm baseon which a financial model can be built and uti-lized effectively by all parties involved in the deci-sionmaking process.

Managers and scholars now approach budget-ing as a process that relies on shared informationand communication for its success; thus, it can-not be easily reduced to mathematical formulae.Despite the plethora of financial and technologi-cal forecasting models that have been introducedin the last 15 years, managers have been reluc-tant to replace qualitative measures with strictly

“W’lnkotsky, et al., op. cit., pp. 185-187.

quantitative ones. Most managers use a combi-nation of qualitative and quantitative techniques,depending on the stage of research of the project.Models are used mainly to explore policy alter-natives. Qualitative evaluation techniques workbest at the level of basic research. A mixture ofquantitative and semi-quantitative techniquesworks well at the level of applied research. Reli-ance on strictly quantitative techniques greatly in-creases when a project enters the last stages ofproduct development.

Many executives shy away from using formalanalytical methods because the data generated bythese models often do not reflect assumptions thatare shared throughout the organization. For ex-ample, Robert L. Bergen, Jr,, manager of cor-porate R&D at Uniroyal, is “leery about individ-uals becoming so committed to a number thatthey will find it hard to reassess a project lateron, adding that there is always a problem whenthe boss ranks things one way and a subordinateranks things another. “44

The degree to which executives and their sub-ordinates’ views are commensurate reflects thelevel of communication between individuals anddivisions within the corporation. Most of the liter-ature in the past 5 years points to gaps in com-munication as the most important difficulty thathas to be overcome if strategy and technology areto be linked. New corporate efforts to mesh strat-egy and technology are occurring at a time whenscientific and technological information is grow-ing exponentially. Management wants informa-tion from corporate scientists and engineers andto share this information with representativesfrom the marketing and manufacturing divisionsas often as possible.

“\lichael F. Wolff, “Selectlng R&D Prolects at L1nlro>’al, Re-search Alanagement, November 1980, p. 8

.

Chapter 5

Government Decisionmaking

Chapter 5

Government Decisionmaking

To be useful to Federal policymakers, quantita-tive methods for evaluating research and devel-opment (R&D) must provide reliable results andfit with existing decisionmaking procedures. Aswe have seen, neither the economic rate-of-returnmodels nor the noneconomic science indicatorscan answer all the questions facing policymakers.The economic models do not even meet the needsof industrial private managers, for whom eco-nomic payoff is the primary concern. One shouldnot be surprised, therefore, that these models of-fer little help in making Federal R&D decisions,in which economic payoff is only one of manycriteria and often a secondary consideration. Inaddition, the users of Federal R&D are not cap-tive. Information produced through federallyfunded R&D is, in most cases, available to any-one who seeks access. Thus, the benefits are dis-persed in a way that makes accounting for themnearly impossible.

The goal of federally funded research is notprofitability, but a means of achieving social ob-jectives, whether they be health, national secu-rity, or the enhancement of knowledge and edu-cation. The Federal research infrastructure isdesigned to provide a stable environment for thesegoals, despite a changing political environment.This creates an R&D management environmentvery different from industry, where reorganiza-tion is more easily achieved.

In addition, Federal research programs must beresponsive to many more groups than industrialresearch efforts, and this affects the manner bywhich the research agenda is shaped. The proc-ess of obtaining funds from the taxpayer for mis-sion research is complex and quite unlike the R&Ddecisionmaking apparatus found in industry. Thebudget process is the first of many hurdles, fol-lowed by levels of decisionmaking at the agencylevel, institute or directorate level, program level,and advisory board level. On occasion, Congresshas attempted to influence the administration andexecution of research programs by using mecha-

nisms such as appropriations riders. Thus, researchfunding decisions in the Federal Government aresubject to levels of review and requirements foraccountability unheard of in industry.

To understand the limited utility of quantita-tive methods for measuring return on FederalR&D, it is important to recognize the complex-ity of the processes leading to the actual invest-ment decisions. Attempts at evaluating researchdecisionmaking should be analyzed in the contextof the scale and structure of scientific activity inthe United States. It is estimated that the FederalGovernment will be responsible for 49 percent ofnational R&D expenditures in 1986, up from 46percent in 1 9 8 2 .1 The structure of support ispluralistic and decentralized, with 10 agencies re-sponsible for R&D functions. The budgets of eachagency differ enormously, as depicted in table 11.Methods for project selection and program evalu-ation also differ between agencies, reflecting thedecentralized and pluralistic nature of the system.These differences are attributable to the age of theagency, the size of the budget, the levels of basicand applied research, agency mission, and themanagement “traditions” institutionalized overtime. In all cases, decisions are made incrementally.

To understand how quantitative methods canbe used in Federal decisionmaking, we have tolook at the types of decisions that must be madeand how these decisions are being made now. Pol-icymakers must establish priorities among all gov-ernment programs, among the various scientificdisciplines, and among projects within a dis-cipline. These priorities are then applied to deci-sions about research budgets, project selection andtermination, and program evaluation. We willlook at how these decisions are now made andevaluate the potential for using quantitative meth-ods to assist in the process.

‘FJat]ona] Science Foundation, Divlslon ot Science ResourcesStudies, Sc]ence and Technology DJta BOOA (P\’ashlngton DC. NSF,1Q86),

61

62

Table 11.- Federal Obligations for Research and Development by Character of Work and R&D Plant:Fiscal Years 1984=85 (thousands of dollars)

Research

Total R&D and AppliedFiscal year and agency R&D plant Total R&D Basic research research Development R&D plant. - - Fiscal year 1984 (estimated):Total, all agencies . .Department of Agriculture. . . . : : :D e p a r t m e n t o f C o m m e r c eD e p a r t m e n t o f D e f e n s e .Department of Energy a .Department of Health and ‘Human Servicesb :Department of the Interior . . . .Department of Transportation . . . . . . . .National Aeronautics and Space AdministrationN a t i o n a l S c i e n c e F o u n d a t i o n . . . .V e t e r a n s A d m i n i s t r a t i o nOther agencies . . . . . . . . . . . . . . . . . . . . : : :: : : :::: : : ::

Fiscal year 1985 (estimated):Total, all agencies. . . . . ... . ... . . . . ...D e p a r t m e n t o f A g r i c u l t u r e . . , . .Department of Commerce . .Department of Defense .D e p a r t m e n t o f E n e r g ya . :Department of Health and Human Servicesb : :Department of the Interior ., ., .,Department of TransportationNational Aeronautics and Space AdministrationN a t i o n a l S c i e n c e F o u n d a t i o nVeterans Administration .,O t h e r S e r v i c e s

46,554,924925,364367,252

27,987,1455,770,6044,921,924

427,558538,429

3,044,4001,247,580

228,1001,096,568

54,072,393926,711282,357

34,510,9846,146,7004,967,872

369,209505,704

3,499,4001,426,567

207,6001,229,289

44,835,777871,942360,021

27,540,0454,825,5764,864,292

421,825515,929

2,888,9001,238,480

220,9001,087,867

52,253,607898,941270,559

34,142,0844,962,2724,953,972

368,989495,204

3,339,4001,414,017

194,5001,213,669

6,981,031386,442

20,522816,590841,671

2,793,052124,667

600689,133

1,172,46615,200

120,688

7,637,587419,727

18,416913,195944,517

2,925,916102,762

400826,721

1,335,80915,000

135,124

8,127,270455,594272,644

2,168,1841,231,7331,705,911

276,33081,990

1,012,03166,014

189,700667,139

8,396,633449,981201,187

2,408,2041,268,9641,679,147

248,55679,630

1,088,06378,208

160,000736,693

29,727,47829,90666,855

24,555,2712,752,172

365,32920,828

433,3391,187,738

16,000300,040

36,21 ‘3,38729,23350,956

30,822.6852,748,791

348, ! 0917, [ 71

415,1741 ,424,(16

19,500341,852

1,719,14553,422

7,231447,1 co945,028

57,6325,731

22,500155,500

9,1007,2008.701

1,818,78627,77011,798

368,9001,184,428

13,900220

10,500160,000

12,550. 13,100

15.620aData ~flo~n for f~gc~ yaar~ 1956.73 ~tj fl~caj years 1974-76 represent obligations of the Atomic Energy COMmlsSlOn (A EC) and the Energy Research and Development

Administration, respectwely.bData shown for fl~al years Igs5-78 represent obligations of the Department of Health, Education, and Welfare.

SOURCE: National Science Foundation.

THE R&D BUDGETARY PROCESSThe process of budgeting is a process of com-

promising among competing values over howfunds should be expended. Since funding is essen-tial for any public policy, the budget process dealsdirectly with how values are allocated in a politi-cal system. ” It is a political process. z

Most descriptions of the Federal budget proc-ess include schematic diagrams (see figures 3 and4) that show the timetable for executive and legis-lative action. These outlines usually highlight thedeadlines for agency budget estimates and the pas-sage of resolutions. The truly significant charac-teristics of the budgetary process, however, areobscured by the arrows and dotted lines. Theprocess is too complex to be characterized solely

by a list of important dates. Outlines and dia-grams do not explain how Congress, the Congres-sional Budget Office (CBO), the President, the Of-fice of Management and Budget (OMB), and theexecutive agencies formulate recommendations forappropriations and budget outlays. No drawingcan adequately represent the influence of the in-cremental method—the major method for calcu-lating budgets at the Federal level.3

Incrementalism informs all aspects of decision-making in normal budget years. Last year’s budgetis the single most important factor in determin-ing this year’s budget, which is the single mostimportant factor in determining this year’s author-

‘VVil]iam L . M o r r o w , Pubfjc Admjrrlstratlon: Po]ltlcs, Pojlcvand the Politica/ System (New York: Random House, 1980), p. 309.

‘Aaron Wildavsky, The Po/itIcs oi the Budgetarv Process ( Bos-ton, MA: Little, Brown & Co,, 1984), p 13

.

63

Figure 3.— Formulation of the Pesident’s Budget

The PresidentApproximate

timing

Budget policy develo

March(or earner in

some agencies)

1

AprilMay

May

June

DecemberJanuaryFebruary

} 4

aln cooperation with the Treasury Department and the Council of Economtc Advmers.

SOURCE Willlam L Morrow, Public A~rrrlrrfsfratfon” PoIItIcs, ~olIcY and fhO F’olltlcd system (New York Random House. 1980), PD 27.28

64

Figure 4.—The Congressional Budget Processm

June

July

1

Budget committees prepare second concurrentresolution and reportAugust

Fiscal year begins

SOURCE; Willtam L. Morrow, PuLWc Adrrrmstraoorr. ~olltics, ~oltcy and the ~olfticd System (New York: Random House, 1980), pp 40, 43

October 1: Fiscal year begins [sec. 501] ?

31: Joint Economic Committee reports analysis of currentservices budget to budget committees [sec. 605(b)] +

Approximately last week of month: President submitsbudget (15 days after Congress convenes) [sec. 601]January

March

April

May

SOURCE: House Budget Committee Section numbers are from the Congressional Budget and Impoundment Control Act of 1974

66

The number of actors involved in the settingof an agency budget is enormous. The three tracksof the budgetary process—authorization, appro-priation, and reconciliation-each focus on differ-ent dimensions of the agency budget, and eachyields its own version. Disparity may occur be-tween the three versions within one Chamber ofCongress, as well as between the House and Sen-ate versions. Throughout the congressional bud-get process, industrial research organizations, sci-entists, and scientific and professional societies,whose members benefit from research funds, lob-by Congress to support increased funding forthose programs.

PROJECT SELECTION

Once an agency receives its budget, projectselection procedures and styles differ betweenagencies. In National Science Foundation (NSF)and National Institutes of Health (NIH) investi-gator-initiated basic research grant programs, theideas for new projects come largely from the sci-entific community through the grant applicationprocess, consensus conferences, and workshops.The National Science Board and the study sec-tions of NSF, along with the Advisory Councilsof the 11 National Institutes of Health, provideadditional overall guidance on agency and pro-gram direction. There is a high degree of confi-dence in the peer review process at these agen-cies and in the scientific community. Questionsabout the perils of peer review persist but therehave been no proposals for change convincingenough to overhaul the system. Until a reliablereplacement is found, qualitative, judgmental ap-proaches will dominate in the selection of basicresearch approaches.

This qualitative approach to project selectionhas been standard in many agencies since Vanne-var Bush recommended it in his 1945 report, TheEndless Frontier. 7 The first R&D agency to trulyimplement the concept of peer review as the meth-od for investing in basic research was the Office

“Vannevar Bush, Science–The Endless Frontier a report to thePresident on a Program ior Postwar Sclentlhc Research (Washing-ton, DC: National Sc]ence Foundation, 19801 (reprinted from Ot-fice of Sc]entlflc Research and Development, 1~4.s).

The Federal budget process, while susceptibleto confusion and manipulation, is a legitimate at-tempt to foster some kind of consensus betweenlegislative and administrative budget actors andbetween competing national policies. In addition,the budget process has become a multi-purposevehicle for political and policy statements that arenot necessarily related to the agency’s missiondirectly. The process is a cobweb of interactionrather than a linear progression from investmentto output.

of Naval Research (ONR), a research agency ofthe Department of Defense (DOD). ONR uses apeer review process that relies on both in-houseand external review, The old ONR model of sep-arating the mission of basic science from the prac-tical mission of the agency provided the modelfor peer review at NSF.

In comparison, mission-oriented programs inagencies such as DOD, the Department of Energy(DOE), the National Aeronautics and Space Ad-ministration (NASA), the U.S. Department ofAgricultural, EPA, and agencies with a relativelysmaller R&D function, such as the Departmentof the Interior, the Department of Commerce, theNuclear Regulatory Commission (NRC), and theVeterans Administration, contract for applied re-search in support of their technology developmentor industry support activities, They tend to re-ceive ideas for new projects from a wide varietyof sources: industry, Congress, their own programstaffs, the national laboratories, and the scientificcommunity. Regardless of the source of a newidea, agency staff will usually conduct a feasibil-ity study to determine whether the new conceptis likely to meet cost, performance, and user-acceptability criteria. If the results of the studyare promising, the program manager will proposethe project as a line item in the new fiscal yearbudget. The administrative officer in charge of theprogram area (often an assistant secretary), thehead of the agency, the examiners at OMB, and(if it represents a sizable fraction of the program

6 5

izations and appropriations. Many items in thebudget are simply reenacted every year unlessthere is a special reason to challenge them. In addi-tion, long-range commitments have often beenmade and the current year’s share for previouscommitments must be taken out of the total andincluded as part of the annual budget.4 These com-mitments preclude comprehensive assessments ofany agency’s budget. Thus, actors in the budgetprocess are concerned with relatively small incre-ments to an existing base. Their attention is fo-cused on a small number of often politically con-troversial items over which the budget battle isfought. Understanding the nature of these battlesis crucial to comprehending the entire process.

The inherently incremental nature of the bud-getary process precludes in-depth, systematic re-views of programs and agencies. In the past 20years, two major attempts have been made to in-fuse some “rationality” into the process, to“change the rules of the game. ” In 1965, Presi-dent Johnson announced the implementation ofthe Planning, Programming, and Budgeting Sys-tem (PPB) at the Federal level. PPB provided deci-sionmakers with data from systems analysis, cost-benefit analysis, program budgeting, and cost-effectiveness studies to support decisions aboutalternative courses of action. However, all of thedata generated did not enable policymakers toestablish priorities in a more systematic fashion;the program failed to account for the political na-ture of the budgeting process:

PPB . . . could determine, within a reasonablemargin of error, what the results would be ifmoney was spent for x instead of y. It could alsoproject how much of x and how much of y couldbe purchased or developed for a specified amountof money. What it could not do was to determinewhether it was best to allocate funds for eitherprogram x or y.5

PPB failed because it set out to tackle an impos-sible task: the goal of its supporters was to “ob-jectively determine what is inherently ideal, ra-tional, and moral in public policy.’”

‘Ib]d.Nlorrow, op. cit., p. 3 0 9

“lbId., p. 310.

In 1974, Congress enacted the CongressionalBudget and Impoundment Act of 1974 (PublicLaw 93-344) to provide more focus on the “bigpicture” of the Federal budget. The two standingbudget committee and CBO determine the appro-priate levels of revenues and public debt for eachfiscal year and the subsequent level of total bud-get outlays and authority. This attempt at bud-get reconciliation has not affected the incrementalnature of preparing separate agency budgets orappropriations bills.

President Carter offered another plan to freethe budgetary process from the constraints of in-crementalism. He introduced the concept of zero-based budgeting (ZBB). Although its applicationis a complicated process, its basic purposes andprocedures are relatively simple to comprehend.Agencies are directed to bracket their programsinto “decision units. ” Each of these units is as-signed a priority status— i.e., the degree to whichit is essential to each agency’s operations. A min-imum expenditure base is supposed to be estab-lished by the agencies to represent “essential” pro-gram obligations that, therefore, are safe frombudget cuts. Theoretically, this is the zero base,with all unnecessary expenditures eliminated. Inpractice, the base was often much higher thanzero. Agencies had a vested interest in protect-ing certain programs. By increasing the level ofthe base, the agencies effectively decreased ZBBseffectiveness in evaluating programs. This new,arbitrarily established base only served to increasethe budget officials’ dependence on the incre-mental method. Less than 2 years after it was in-troduced, ZBB was abandoned by all Federalagencies, with the exception of the EnvironmentalProtection Agency (EPA), which still employs thismethod today.

Incrementalism in the budgetary process is onlyone difficulty encountered in attempting system-atic review of research agency programs. Com-prehensive review of Federal R&D efforts is fur-ther complicated by the fact that the FederalGovernment does not have a separate R&D bud-get. Federal funding for R&D is the sum of thoseprogram requests submitted by individual agen-cies to OMB, subsequently by the President toCongress, and approved, rejected, or altered dur-ing the budget review and appropriation process.

6 7

budget) the staffs of the authorizing and appropri-ating committees in Congress will review the pro-posal. The project selection process is completeonly after a project has been formally includedin a congressionally approved budget.

Within DOD, the Defense Advanced ResearchProjects Agency (DARPA) uses no formal systemof peer review for project selection, but relies onthe management of contractors by DARPA pro-gram managers. Occasionally, the Defense ScienceBoard examines an area of research supported byDARPA in order to make recommendations forfuture action.8

DOE supports R&D carried out through its ownlaboratories and by contracting with universitiesand industry. Research is evaluated for fundingthrough the use of peer review, programmatictechnical review, and programmatic managementreview.

NASA relies heavily on internal and externaladvisory committees for planning future missions,assigning priorities among them, and selecting spe-cific experiments. The NASA Office of Aeronau-tics and Space Technology research programs se-lect projects by collaborative-review by a networkof both researchers and users of research results. 9

Whether project selection is conducted by peersor agency management, the traditional criteria forselection are based on qualitative judgments. Likeindustry, government seldom uses quantitativeproject selection models. A 1974 review of the useof quantitative methods for project selection ingovernment agencies revealed that they were usedvery little except in a few decisions involving largedevelopment projects. Recent surveys reveal noevidence that the patterns and extent of use ofquantitative techniques to evaluate proposed re-search in the Federal Government are undergo-ing any significant change. *O Recent DOE surveysreached similar conclusions. 11

J. David Roessner’s12 limited study of the useof R&D project selection models in DOE offersinsight into the different ways models can be usedin Federal agencies and reveals some of their limi-tations. He reports that the most extensive use ofsuch models took place in DOE’s energy conser-vation programs. Program managers at the En-ergy Research and Development Administrationand later at DOE were under unusual pressure tojustify public expenditures for energy conserva-tion. At the same time, the conservation programwas confronted with mountains of proposals thathad to be screened and acted on in some system-atic, defensible fashion. The models helped screenprojects and select those most likely to pay off.They were also used to justify the program toCongress, where models received considerable at-tention. Congress cared less about the project-by-Project scores than about the aggregate benefit .scores achieved by all projects funded by a pro-gram. Quantitative estimates of oil displacementand energy savings generated by the models provedextremely useful in defending expenditures for R&Dprograms. One successful model compared cost-benefit ratios at the program level based on oil sav-ings with and without Federal expenditures. Thesemodels evolved gradually and were applied overseveral years to project screening and budget deci-sions. DOE usually used the models with appliedresearch programs, in which the links to commer-cial applications were apparent. 13

The Department of Energy’s use of cost-benefitanalysis for its conservation program is not rep-resentative, The connection between technicalimprovements and economic benefits in energyequipment is much more straightforward than forother technological advances, making a predic-tive model somewhat useful. Also, economic ben-efits were the primary goal of the program. Suchconditions are rare in Federal research programs,and it is instructive that even DOE has limitedits use to a few applied research programs.

‘J.M. Logsdon and C.B. Rubin, Federal Research EvacuationActivities (Cambridge, MA: Abt Associates, 1985), p. 14.

‘Ibid., p. 17.‘“H. Lambnght and H. Sterling, A/ationaf Laboratories and Re-

search Evaluation (Cambridge, MA: Abt Associates, 1985).“K. G. Feller, A Review of Methods for Evacuating R&D (L1ver-

more, CA. Lawrence Llvermore Laboratory, 1980); and J. DavidRoessner, R&D Propxt Selectlon Models In the U.S. Departmentof Energy (Atlanta, GA: Georgia Institute of Technology, 1981),

‘ : Roessner, op. cit.‘Ibid., pp. IV-1, 2; V-2.

68

RESEARCH PROGRAM EVALUATIONQuantitative methods are only occasionally

used in evaluating the productivity and relevanceof existing programs. A number of agencies areexperimenting with quantitative techniques, butfew have adopted them for use in systematicevaluation of research programs. Several recentsurveys of research managers in the agencies andnational laboratories found little evidence ofquantitative techniques in research evaluation.

NIH is the only agency consistently using quan-titative evaluation methods for accountability andprogram planning. Its bibliometrics analysis ef-fort is described in chapter 3. The Alcohol, DrugAbuse, and Mental Health Administration is plan-ning to use bibliometric approaches in evaluat-ing research programs in the future. For otheragencies and other techniques, we find a historyof disappointment.

The National Bureau of Standards (NBS), forexample, made an abortive attempt to measurethe economic impact of individual projects in theSemiconductor Technology Program. NBS askedfirms that subscribed to a program publication toestimate the benefits of each project to the firmand the costs of implementing technical informat-ion received from NBS. Agency analysts thencompared the social costs with the estimated so-cial benefits. They also used a production func-tion to measure the productivity of the entire pro-gram. The objective was to estimate the changesin a firm’s productivity attributable to changes inthe stock of R&D capital generated by NBS. NBSstaff reports that these studies were discontinuedbecause of serious theoretical and methodologi-cal problems. 14 NASA’s macroeconomic and ma-croeconomic approaches to measuring the effectsof its R&D programs (see ch. 2) also met with seri-ous criticism, and NASA discontinued its efforts. 15

The Department of Energy employed an elab-orate, quantitative evaluation scheme based on

peer review to evaluate its Basic Energy ScienceProgram in the early 1980s. Forty small reviewpanels used a formal rating sheet to evaluate 129randomly selected projects on publications produced, personnel achievements, and project summary descriptions. Panel members rated projectsfor researcher quality, scientific merit, scientificapproach, and productivity. The evaluators com-pared the results with the scores of comparablyfunded, nonlaboratory projects also rated by thepanels. DOE has not applied this expensive andtime-consuming evaluation method to other pro-grams, but the ONR has adopted some aspectsof the technique.l6

In 1982, a National Academy of Sciences panelconducted a study for NSF of approaches to eval-uating basic research. They concluded that, be-yond peer review, “any additional evaluation pro-cedures should be introduced only if they clearlyenhance rather than constrict the environment inwhich the research proceeds, and that formal tech-niques cannot usefully replace informed techni-cal judgment. ”17 A 1984 survey of 41 researchmanagers in 11 Federal agencies found that non-quantitative methods dominated evaluation:

Some form of peer review was used by almostevery Federal agency, both for selecting individ-ual or team research projects and for exercisingmanagerial control over them. Peer review is alsothe major way that agencies build a case to dem-onstrate the value of research they support. 18

No Federal agency “has in place a research evalu-ation system which appears to move substantiallybeyond the organized use of “informed technicaljudgment .“19

A review of national laboratory evaluationtechniques uncovers a similar picture. 20 Mostevaluations of laboratory research are relativelyunstructured and do not assume major importanceamong laboratory activities. 21 The complexity of

“Logsdon and Rubm, op. cit., p, 34.“Ibid., p. 15; and Henry R. Hertzfeld, “!vleasurlng the Eco-

nomic [mpact of Federal Research and Development Investments]n C]v~lIan Space Act]v]t]es, ” paper presented to the National Acad-emy of Science Workshop on “the Federal Roie In Research and De-velopment, ” Nov. 21-22, 1985, pp. 9-12 and 16-21,

‘OLogsdon and Rubin, op. cit., pp. 26-28: and Lambrlght ancStirling, op. cit., p. 28.

‘Logsdon and Rubin, op. cit., p, 38,‘Ulb]d., p. 25.‘Ibid., p. 38.‘“Lambrlght and Stirling, op. c]t,2) Ibid.

69

lab roles, which include research performance,research management, and entrepreneurship, pre-cludes most formal, quantitative evaluation tech-niques. 2 2 s o m e l a b o r a t o r i e s o c c a s i o n a l l y u s e s t r u c -

tured peer review and bibliometric techniques,often performed by outside contractors, but labmanagers view these as “supplements to the lessstructured evaluations, rather than substitutions. ”23

Although economic models for R&D perform-ance came into use in the 1950s, government re-search managers have not adopted them for pro-gram evaluation. This reflects the nature ofresearch manager concerns as well as the accuracyof the models. Research managers are responsi-ble primarily for the quality of research in theirprograms, and economic payoff does not neces-sarily reflect the quality of research. A break-

‘Ibid., p. 9.‘Ibid., p. 10. .

through in basic or applied government researchdoes not guarantee an economic benefit. No eco-nomic effect will occur unless a private companydecides to incorporate the breakthrough in a prod-uct, and the success of that product depends onsuch factors as the availability of capital, effec-tive product development, consumer interest,marketing skill, tax and regulatory environment,and competition. Research managers have no sayin these other factors and no control over the com-mercial uses of the research they manage. Theytherefore limit their attention to what they cancontrol—the quality of research.

Bibliometric methods offer a quantitative meas-ure of research quality, but one not reliableenough to serve as the sole basis of research evalu-ation. These methods are, however, a useful sup-plement to informed technical judgment and thepeer review process.

FORECASTING AND STRATEGIC PLANNING

The extent of agency use of strategic planningto identify promising future directions for civil-ian research is not known. DOE’s Energy ResearchAdvisory Board, NSF’s National Science Board,and NIH’s scientific advisory councils provideguidance that might pass for strategic planning.The NRC and its constituent bodies,—the Na-tional Academy of Sciences, the National Acad-emy of Engineering, and the Institute of Medi-cine—carry out numerous reviews of researchprograms and research fields for the executive andlegislative branches. The most comprehensive ofthese are the Research Briefings and Five YearOutlooks prepared by the Committee on Science,Engineering and Public Policy. Since all NRCreports are prepared by committees of scientists,they represent, to some degree, the informed,consensus-based peer judgments of the scientificcommunity on the state of the research enterprise.None of these, however, can be said to consti-tute true strategic planning or forecasting. For ex-amples of more systematic forecasting and plan-ning activities related to science and technology,one can look to Japan.

Planning for Innovation in Japan

The Japanese Science and Technology Agency(STA), established by the Japanese Governmentin 1956, has relied on strategic planning for itssuccess in identifying technological innovations.In 1969, STA’s responsibilities expanded to in-clude funding research as well as coordinating re-search activities among the various governmentministries and agencies. In addition, STA beganto make technological forecasts. After the resultsof the first study were obtained, the agency real-ized these forecasts would be vital to the creationof rational, long-term research policies and madethem a regular part of its operating procedures.

The forecast effort begins by identifying eco-nomic and social needs. Forecasters then survey

research areas to identify potential scientific andtechnological developments that can meet theseneeds. They then establish priorities for variousR&D plans. Each forecast is presented in twoways: one that is “exploratoy or predicative, re-lating to individuals’ expectations of change giventheir accumulated knowledge and experience: and

70

another that is normative— that involves settingan objective and a time-scale within which it isto be achieved. ”24 The forecast is based on a sur-vey of over 2,000 people from government, acade-mia, and industry with a broad knowledge of sev-eral scientific and technological research areas,who not only answer questions but who also com-ment on their colleagues remarks.

Industrialists, academics, and government offi-cials have cited four benefits from the surveyprocess:

1.

2.

3.

4.

The studies provide a mechanism to ensurethat researchers in all sectors, along with pol-icymakers in government and industry, areperiodically forced to think systematicallyabout the long-term future.The forecasts yield a general summary ofwhat is happening, or likely to happen,across the entire range of R&D activities.They therefore permit more “holistic” vision-making, enabling the potential longer termcross-impacts of developments in ons re-search field on another to be identified at anearly stage.By surveying comprehensively the intentionsand visions (and thus indirectly the currentstrategic R&D activity) of the industrial re-search community, the surveys provide auseful mechanism for synthesizing major re-search trends across science-based sectors.. . . It is the existence of such surveys thatis in part responsible for the strong agree-ment among Japanese firms as to what arelikely to be the critical future developmentsin their sector.The STA forecasts provide a useful mecha-nism for helping government establish na-tional priorities in allocating resources. Re-quirements for infrastructural support canbe identified from the “bottom-up” by indus-try, rather than being imposed by state plan-ners who may not always be in touch withindustrial problems. Although such forecastsdo not, in themselves, lead directly to pol-icy decisions, the systematic informationwhich they generate helps narrow down the

“John Irvine and Benjamm R. Martin, Foresight m Science:PicA]ng the Winners (London: Frances Pinter, 1984), p, 108.

range of different views that can be held ona particular R&D related issue, bringingeventual consensus that much closer. 25

In addition to the STA, the Ministry of interna-tional Trade and Industry (MITI) and the Agencyfor Industrial Science and Technology (AIST) areresponsible for providing funding and long-termguidance for applied research and development.Each MITI division, representing a major indus-trial sector, establishes a long-term plan, whichis revised every 3 to 5 years. Officials from boththe ministry and the agency try to incorporatethese plans into MITI’s vision of the future of Jap-anese industry. This vision serves as the basis forMITI and AIST’s long-term R&D plan, which isalso revised every 3 to 5 years. This plan enablesthe ministry to spot research trends early and topredict how new technologies might develop.These predictions, in turn, form the basis for anR&D policy that concentrates on initiating re-

search programs in areas of strategic importance.Like STA’s forecasts, MITI’s visions are based onsuggestions that come from the bottom up. Irvineand Martin stress that the myth of “Japan, Inc. ”has arisen because foreign commentators haveoften overlooked this point:

The process involved here is not one of central-ized “top down” planning by MITI, which thenimposes its objectives on industry and others (whodo not question them), Instead, most influencetends to flow in precisely the opposite direction,with MITI’s role largely confined to “tapping into”the views and firms where consensus lies. Onlyas a last resort are priorities imposed—for examp-le, to give one industrial sector’s agreed programprecedence over another’s.26

Before the government introduces a new policymembers of an informal working group, whichrepresents a major trade association, meet to dis-cuss common long-range goals. These member:are employees of firms that continuously moni-tor R&D developments throughout the worldThey also have access to the company’s internaforecasts of technological innovations. The informed comments of these members are used byconsulting groups when they design surveys of in

“lb[d., pp. 110-111.‘eIbid., p. 118.

71

dividual industrial sectors .27 The working groups’suggestions and the sectorwide surveys providean essential link between STA’s and MITI’s mac-roforecasts and firms’ internal forecasts for spe-cific products .28 According to Martin and Irvine:

From the point of view of industry, the sectorforecasts, because they are much more specificthan the macroforecasts, are much more valuablefor planning corporate R&D strategy. Equally, thesector studies, based as they are on a synthesisof industrial views, constitute a key input into dis-cussions within MITI and STA.29

MITI’s role is to construct the broad frameworkwithin which a consensus on long-range R&Dgoals can be established; to catalyze the forma-tion of consensus by sorting and publicizing theresults within the relevant sectors; and to try tobuild a consensus among the various industrialsectors as to long-term R&D priorities.

Several lessons can be learned from MITI’s con-sensus-generating approach to strategic researchforecasting. First, forecasts that successfully iden-tify research areas of long-term strategic impor-tance are based on up-to-date background infor-mation on research trends gathered from industry,academic, and government reports from aroundthe world. Second, the forecasts incorporate“technology-push” and “market-pull” perspectivesbecause scientific and technological advances mustbe coupled with changing market demands fortechnological innovations to be successful. Third,there are a number of advantages that can begained from adopting a bottom-up approach toforecasting rather than a centralized, “top-down”approach.

Apart from being dependent on a narrow rangeof information inputs, “top-down” forecasts andthe resultant research policies are more likely toantagonize not only the basic science community(which may feel that it has been inadequately con-sulted in the forecast process), but also industry(which naturally tends to feel that it is in the bestposition to judge the commercial prospects forstrategic research). ’o

The last and perhaps most important lesson tobe gained from the STA and MITI forecasts is thatthe process of generating the forecasts is muchmore important than the product—the specific re-sults they yield. The process unites people fromdifferent groups and different professions withinthose groups and provides a framework withinwhich they can “communicate directly or indi-rectly (through a Delphi-style forecast) with eachother. “31 Policymakers, professional forecasters,scientific analysts, and academic and industrialresearchers are periodically forced to think aboutlong-term R&D activities by the process. This ena-bles them to coordinate research plans and to forma consensus on priorities for future strategic re-search. Furthermore, the process generates a feel-ing of commitment to the outcomes of the fore-casting studies. Thus, the predictions becomeself-fulfilling prophecies. The Japanese contendthat these five C’s—communication, concentra-tion on the future, coordination, creation of con-sensus, and commitment—have benefited theirstrategic planning efforts tremendously. Untilnow, these benefits have outweighed such dis-advantages as forecasts’ tendency to encourageconservativism and breed excessive competition.Martin and Irvine warn that this balance mightbe upset in the future “as the Japanese place in-creasing emphasis on more basic research (wherecreativity and unconventional approaches areclearly at a premium ).”32

While other students of Japan warn againstplacing too much faith in the apparent tidinessand completeness of the framework building proc-ess, the importance of wide participation in goal-setting is apparent.

The centrally coordinated Japanese R&D sys-tem has served Japan well in applying basic re-search findings. Yet, pluralism in the U.S. R&Dsystem encompasses several attributes. In testi-mony before the Task Force on Science Policy,Rodney Nichols of Rockefeller University statesthat the pluralism of the system “hedges against

“Ib id , , p 301‘ @ Ib]d,, P 129‘ I b i d . , p, 128‘Ibid., p. 143

——‘lbId., p. 144,

“Ibid.

72

fluctuations in the fashions and policies influenc-ing any lines of R&D support. ”33

A full-blown pluralistic system depends upona high level of sustained R&D. It runs the risksof some redundancy when sponsors overlap theirsupport—surely at the research end of the spec-trum, where there are many, small projects under-way through many sponsors. By doing so, it gainsthe long-run advantages of giving all missions awindow on research.

Pluralism also protects against the inherentfrailty, even occasional ignorance, of decisionsby research managers. It aims, in principle, tostrengthen the broad swath of R&D by beingaware of how unpredictable are the origins ofgreat ideas: and how unpredictable are the con-sequences of results that first seem mere curiosi-ties. Thus, some funds go to all good ideas in or-der to ensure that the few seen later to be the besthave had a chance.34

“Rodney W. Nichols, testimony presented before the U.S.House of Representatives, Committee on Science and Technology,Task Force on Science Policy, Oct. 23, 1985. “Ibid, pp. 10-11.

0

. ,