Bill ValdezDirector, Planning and AnalysisSeptember 2005Email: Bill.Valdez@science.doe.gov

U.S. Department of EnergyU.S. Department of EnergyOffice of ScienceOffice of ScienceBudget and PlanningBudget and Planning

Defining the Problem

“How much should a nation spend on science? What kind of science? How much from private versus public sectors? Does demand for funding from potential science performers imply a shortage of funding or a surfeit of performers?”

To answer these and related innovation policy questions we need agreement on:

• Terms & Conditions – Fed vs. private investment– Definition of “value”

• Level of Analysis– Project/program/agency/systems– U.S./International

• Resources vs. Expectations– Manhattan Project or After School Project– This year, 2010, or when fusion energy goes

commercial?

R&D Evaluation – Current State

Evaluation Types:

•Peer Review

•Output Metrics

ProjectsProjects ProgramsPrograms PortfoliosPortfolios OrganizationsOrganizations SystemsSystems<$5M

~$200M~$3B

~$25B

~$300B

Evaluation Types:

•Committees of Visitors

•Output & Outcome

Metrics•Case

Studies•Randomize

d Trials

Evaluation Types•National Academy

reviews•Econometric

Modeling•Committee Reviews•Case Studies

Evaluation Types:•Advisory

Committee reviews•Econometric

Modeling•Risk/Options

Modeling•Case Studies

Evaluation Types•NAS/COSEPUP

International Benchmarking

•Longitudinal Studies•Innovation Indexes•Case Studies

Evaluation Results– Current State

Evaluation Results:

•Cost•Schedule•Milestone

s•Data

Rates•Human

resources•Quality &

Relevance of project

ProjectsProjects ProgramsPrograms PortfoliosPortfolios OrganizationsOrganizations

Detector MIE at RHICTPC $5M

Nuclear Physics$400M

Office of Science$3.5B

U.S. R&D ~$312B

Evaluation Results:

•Characterize quark-gluon plasma

•New PhD’s produced

•Improve facility operations

Evaluation Results:•Achieve scientific

breakthroughs•Meet Administration

goals•Improve overall

management efficiency

•Advance energy efficiency

•Nobel Prizes

Evaluation Results:

•New scientific directions

•Meet DOE goals•Improve scientific

workforce•Increased

operational efficiencies

Evaluation Results:

•Increased Life Expectancy

•GDP Growth•New Knowledge•Increased National

Security•New Industries•Energy

Independence

Department of Energy

$23.4B

“Systems Level Analysis”

“Systems Level Analysis” Asks & Answers Fundamentally New

Questions:

Is it possible to compare the outcomes of one innovation system to another?

• Universities vs. National Labs?• Public vs. Private?• NSF vs. SC?• Applied vs. Basic R&D?• Japan vs. EU?

Would comparisons of performance of systems lead to:

• Management efficiencies?• New policies?• Greater funding for R&D?

Systems Level Outcomes

Examples include:• The 3% Solution• Regional Fallacies• Paper Chases• Predicting Prestige

(a.k.a. fortune telling)http://www.oecd.org/

http://www.compete.org/

Existing “Innovation Indexes” suffer from a host of problems, primarily a lack of context, causality, and comparability.

System Performance

Highly performing innovation systems should have the following attributes:

•Competition for Resources– (Money, Ideas, People, Facilities)

•An open market place for ideas– (Patents, Papers, Copyrights, IP)

•Resources sufficient for system growth– (People, Equipment, Money, Land, Energy)

•Checks & Balances– (Transparency, Multiple Funding Sources, External

Review)

An absence of any of these will seriously impair the effectiveness & efficiency of any innovation system.

Pre-Conditions for Systems Level Analysis

Before you can start, you need thefollowing:

1) Knowledge of who/what you are analyzing; i.e., the “system” that you are analyzing.

2) A review of past efforts.3) A working theory.4) An answer to the data challenge.5) New tools and methodologies.6) Partnerships to share the cost, set

comparable standards, and bring new ideas to the table.

Innovation Fits within a Dynamic System

Figure 1. National Innovation EcosystemFigure 1. National Innovation Ecosystem

TALENT•World Class Innovators•Adaptable Workforce

•Science & Engineering Skills•Magnet for Global Talent

Innovation Demand•Macro Demand

•Consumer•Business•Government

•National Priorities•Market Access•Industry Structure•Technology Diffusion•Standards•Profitability•Stock Valuation

INVESTMENT•Valuing long term innovation•Multiple disciplinary research

•Early stage investment•Service sector innovation

Innovation •Policy

•Strategy•Process•Insight

Accelerate level, quality, efficiency and profitability of US innovation

(overall success metrics)

Growth, Jobs, Standard of Living, Wealth, Comparative Advantage

INFRASTRUCTURE•World-class infrastructure•Innovative public sector

•Regulatory and legal system•21st Century IP system

Innovation Inputs•Creativity•Research•Knowledge •Information

Review of Past Efforts

We have worked for the past six years to develop a body of knowledgethat will inform our efforts:

• A review of theory, evaluation and management literature

• Partnerships with the private sector (IRI, Council on Competitiveness, Santa Fe Institute) and academia (GWU, USC, etc.).

• Ongoing interactions with the evaluation community through FedEval, AAAS, AEA, & WREN.

Theory-Based Analysis

An absence of theory hinders any effort to develop systems-level evaluations.We are developing:• Management/Economic/Social Science Theory

– What is the “net benefit” of public investment?– What differentiates Federal R&D agencies from

Bell Labs and Microsoft?– Is there such a thing as a “knowledge multiplier”?

• Evaluation Theory– What is the definition of “value”?– How can the innovation system be characterized?– What are the appropriate research questions?“All theory depends on assumptions which are not quite

true.” Robert Solow, 1957

Some Emerging Theories

We are beginning to see the emergence of a consensus on theoretical underpinnings that are relevant to Federal agencies.

• Management Theory– Gretchen Jordan’s “R&D Profiles” work.– DOE management benchmarking study.

• Social Science/Economic Theory– Science as a “co-evolving ecological community” (USC).– Models of knowledge flow (Ventana).

• Evaluation Theory– COSEPUP International Benchmarking Report– Jerry Hage’s “Innovation Systems” book.

Tackling the Data Challenge

The first problem confronting any attempt to do systems-level analysisis the absence of comparable data.There are at least five reasons:

1. R&D data is largely disaggregated.2. Aggregating R&D data is horrifically

expensive.3. Aggregating R&D data is horrifically intrusive.4. “Standard” definitions for R&D data do not

exist.5. Complexity is daunting.

Good Data is a Barrier

Complexity is Daunting– U.S. Economy is $12.2 Trillion, w/50 States &

3,066 counties.– Federal Budget is $2.6 Trillion, w/1,400 Programs– $764 Billion Global R&D investment. $312 Billion

U.S. R&D Investment - $132 Federal plus $180 Industry

– 3,700 degree-granting Colleges & Universities with 15.6 Million Students.

– 329,300 High Tech Establishments employ more than 5 Million High Tech Workers

– 4.7 Million Scientists, Engineers and Technicians – R&D data is typically found in journals, conference,

workshops, pre-print servers, and scientific databases

Sources: OMB FY06 Budget Request - Federal Budget, number of programs, U.S. Economy and Federal R&D American Association of Counties – U.S. counties OECD - Global R&D for 2003NSF Statistical Research Services - Industry R&D in 2003 (projected); U.S. Colleges, Universities and students for 2000, S&E workforce for 2003.AeA - High Tech “establishments” and High Tech Workers for 2003

Full U.S. Patent Office Database• Scientific Research Papers (Journal Papers,

Technical Reports, “Gray Literature,” Websites, Workshop Proceedings, etc.)

• Copyright submissions• International Data (OECD, EU, Japan, Korea, etc.)

Data being generated is huge:• 6 Million DOE records of papers, technical reports, etc.

since 1945• 17 Million hits annually on DOE scientific databases• 2.3 Million abstracts (65,000 new ones/year) from NIH• 670,000 scientific articles published annually in open

literature• 160,000 annual U.S. Patents (6.5 million total U.S.

Patents)

We are collecting vast quantities of information from extremely diverse sources/databases, including:

Data Mining/Data Visualization

IT tools that could be used for S&Tevaluation purposes have progressedover the past decade

• National Security concerns have sparked tremendous innovation in data mining/data visualization.

• Pacific Northwest National Laboratory is a leader in developing new tools.

• SC is exploring the use of these tools for a wide variety of applications and to solve the problem of “information overload.”

“Galaxy View” of 52,000 DOE Researchers (1976-present)

New Policy and Strategic Analysis Tools

The next generation of innovation evaluation tools will create opportunities fornew analyses to assess policy and strategic choices.

• Growth Accounting—economists will be able to better estimate the nation’s productivity performance in terms of contributing factors and outputs.

• Knowledge Economy—composite knowledge indicators will improve investment decisions for R&D, education and capital resources.

• Financial Reporting—financial reports could provide a balanced scorecard of physical as well as intangible assets.

• Valuation of Innovation—business executives and financial markets could better value R&D activity and related intangibles, estimate financial results, improve long term stock market valuations and predict outcomes.

• System Dynamics—expanding the range of “real-time” innovation metrics would help build more robust systems dynamics models and policy simulations. .

• General Purpose Technology (GPT) — improved analysis of the strategic contribution of GPT’s which set the stage for incremental innovation and have the inherent potential for pervasive application in a wide variety of industries.

• Tech-led Regional Development and Clusters—shift the emphasis from strengthening inputs to the innovation infrastructures toward improving the efficiency, rate and output of innovation.

New Tools & Methodologies

Because we are a mission agency, our focus is on tools that will help solve problems. Three general tools are being explored:

• Advanced Bibliometrics/Patent Analysis

• Network Analysis

• Modeling/Simulation

SC Patent Analysis Tool - Data

Abstracts and background data from 12,869 patents originating from work sponsored by DOE• 4,390 SC Patents

Abstracts and background data from 50,263 patents that cite DOE patents as prior art representing 82,737 citations• 27,699 citations of SC patents

1,231 Distinct organizations have attributed one or more of their patents to work sponsored by DOE• 453 organizations attribute to SC

13,345 Distinct organizations have cited one or more of DOE patents as prior art.• 4,172 organizations have cited SC

Distribution of SC-Sponsored Patents

Distribution of Citations to SC-sponsored Patents

Surgical Application of Pulsed Laser Technology

DOE holds 5 patents related to the surgical application of pulsed laser technology:

– 4,381,007, Multipolar corneal-shaping electrode with flexible removable skirt.

– 4,326,529, Corneal-shaping electrode. – 4,686,979, Excimer laser phototherapy for the dissolution of

abnormal growth. – 4,349,907, Broadly tunable picosecond IR source. – 5,720,894,: Ultrashort pulse high repetition rate laser system for

biological tissue processing.

Although most of the underlying research was originally conducted in the early to mid 1980's, these patents continue to generate broad interest within the medical community

To date, these patents have been cited in over 350 patents from some of the world's leading innovators in surgical equipment and techniques.

We are building upon work done by private industry that identifies undervalued companies and develops “non-economic” valuations of research portfolios.

• Technique examines the entire U.S. patent record in combination with scientific papers.

• “Hot” technology areas are identified.

• Individual organizational performance can be assessed and compared (Federal agencies, universities, companies, nations).

• Has the potential to be a predictive tool of performance.

“Technology Hotspots”

Federal Agency PerformancePercentage of Patents Citing Papers Funded by Different

Agencies that are in Hotspot Technologies

3.2%

2.5%

4.0%

4.3%

5.0%

0% 2% 4% 6%

All Agencies

NIH

NASA

NSF

DOE

Pap

er F

un

din

g S

ou

rce

Percentage of Citing Patents in Hotspots

“Successor Patents” Could Prove Predictive

0%

10%

20%

30%

40%

50%

60%

Any of 4Agencies

All Patents DOE NASA NSF NIH

Funding Agency

Percentage of Patents Citing Scientific Papers Funded by Different Government Agencies that Become Next Generation Patents

1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Coherent light generators

Superconductor technology

Drug, bio-affecting compositions

Coating apparatus

Power plants

Measuring and testing

Electricity - measuring and testing

Chemistry - electrical and wave energy

Chemistry of inorganic compounds

Stock materials

Catalysts and solid sorbents

Synthetic resins or natural rubbers

Semiconductor devices

Image analysis

Optics - systems

Surgery

Chemistry - electrical current

Chemical apparatu

Coating processes

Organic compounds

Optics - measuring and testing

Communications

Radiant energy

Chemistry - molecular and microbiology

University (Patents -- Citations)Boston University (14 -- 22)Brown University (18 -- 10)Cal Tech (30 -- 122)Columbia University (10 -- 14)Emory University (6 -- 8)Johns Hopkins University (10 -- 28)Michigan State University (21 -- 46)Northeastern University (6 -- 39)Northwestern University (37 -- 34)Princeton University (8 -- 37)Stanford University (42 -- 21)University of California (1316 -- 689)University of Dayton (20 -- 27)

DOE Support at Research Universities Has Produced Patents and Citations in a Wide Range of Areas*

University of Delaware (18 -- 21)University of Kentucky (5 -- 21)University of Michigan (14 -- 44)University of Minnesota (12 -- 35)University of Missouri (20 -- 19)University of New Mexico (10 -- 17)University of Pennsylvania (6 -- 19)University of Pittsburgh (5 -- 26)University of Rochester (9 -- 23)University of Texas (7 -- 102)University of Utah (13 -- 27)University of Washington (13 -- 35)Yale University (5 -- 16)

*Width of color band indicates relative number of patents in each classification.

Network Analysis – Tracing Ideas

Theory is that science is conducted through networks of people and their interactions are key to understanding the “value” of public investments.

Network analysis builds on:• “Value chain” analysis used by industry

• “Social Network Analysis”

• Simulation & modeling techniques

• Case study techniques

• Patent & paper analysis

Definition of Value/Outcomes

Focusing on “knowledge” as the key outcome of basic research. We are systematically tracking knowledge

outcomes and how they flow.

• Applied Math: First study looked at algorithms, software products, generic mathematical approaches, etc., of SC’s Applied Math Program.

• Nanoscience: Second study, just begun, will look at SC’s five nanocenters and nanoscience evolution as a discipline.

• European Union/FP6: Complementary study, done by European Union, examined how funding affected network formation in Europe.

Initial Results

First results indicate:1. It is possible to assign “value” to

knowledge as it travels through the ideas marketplace.

2. Factors influencing the transfer of knowledge can be identified.

3. We should be able to compare innovation systems and model their behavior.

4. Knowledge multipliers are real.

Emergence of the Term “Nano” in Open Literature*Showing Representative DOE Papers and Patents

*Terms with at least 10 occurrences in at least one year. Width of color band indicates relative number of occurrences‡ Papers identified by the Institute for Scientific Information as among the Top 25 Highly Cited Papers in Nanotechnology.

System Dynamics Modeling for a Dynamic System

What would the models look likeand what will they tell us?• System Dynamics Modeling is an alternative to

Optimization/Econometric Modeling.• Builds on work done by Jay Forrester, MIT, over

past 45 years.• Uses “soft data” and “system attributes” to

model non-linear systems, such as R&D and knowledge.

• Results are “scaleable” and can go from project to program to organization to system.

Where Are We Headed?

• European Commission is cooperating on joint research.

• WREN members are slowly coalescing around the problem.

• International workshops will lead to “bureaucrat-to-bureaucrat” cooperative frameworks.

• Private sector and academia are heavily engaged.

• OSTP will launch an effort.

We need help to leverage scarce resources…and we are working to get that help.