Osamu NAKAMURA, Ph.D. Director for Technology Evaluation

Report about R&D Evaluation Workshop in Japan:

Evaluation of Science and Technology Policy System in Japan: Country Case Study

Osamu NAKAMURA, Ph.D.

Director for Technology EvaluationTechnology Evaluation and Research Division, METI

JAPAN

http://www.mri.co.jp/PROJECT/2005/20051011_ird01.html

Evaluation of Science and Technology Policy System in Japan:

Country Case StudyOrganizers:Japan Society for Science Policy and Research Management (JSSPRM)Ministry of Economy, Trade and Industry (METI)Ministry of Education, Culture, Sports, Science, and Technology (MEXT)National Institute of Advanced Industrial Science and Technology (AIST)National Institute of Science and Technology (NISTEP)… In cooperation with WREN! (Washington?/World? Research Evaluation Network)

Date:June 2-3, 2005

Place:Mita Public Hall, Tokyo, JAPAN

Policy Evaluation Exercises in Seoul / Tokyo

General Discussion in Seoul

(May 30 & 31, 2005)

Country Case Study Exercise in Tokyo

(June 2 & 3, 2005)Project Evaluation

Policy Evaluation

Program Evaluation

In cooperation with WREN!

Session 1: Science and Technology Policy System in Japan moderator: Prof. Kondoh (Yokohama Natl. Univ.) commentator: Dr. Holland (White House, OSTP) Prof. Georghiou (Manchester Univ.) Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in

Japan moderator: Prof. Niwa (Natl. Graduate Ins. for Policy Studies) commentator: Dr. Reeve (EC) Dr. Shipp (NIST) Session 3: Evaluation of Science and Technology Programs and Reflection of its Result in

Japan moderator: Prof. Miyazaki (Tokyo Inst. of Technology) commentator: Dr. Oros (USDA) Mr. Teather (NRC of Canada) Session 4: Evaluation of National Research Institutes moderator: Prof. Hayashi (Natl. Inst. for Academic Degrees and Univ. Evaluation) commentator: Dr. Jordan (SNL) Mr. Valdez (DOE)



Kick-Off:Prof. Vonotas (GWU)

Progress of R&D Evaluation in Japan

Policy

Program

Project

Institute

Ex-ante Monitoring/ Ex-post Impact As.

2002 　Government Policy Evaluations Act

Ryo Hirasawa, 30/08/2005, [email protected]

2001National Guideline on the Evaluation for Governmental Research and Development (II)

2001Law on the General Rules of Incorporated Administrative Agencies

1997 　 National Guideline on the Evaluation for Governmental Research and Development (I)

Nat

iona

l Gui

deli

ne

on t

he

Eva

luat

ion

for

Gov

ern

men

tal R

esea

rch

and

Dev

elop

men

t (I

II)

2005


Session 1: Science and Technology Policy System in Japan

Japanese Science and Technology Policy

System and Evaluation

Dr. Tomohiro IJICHI

Associate Professor, Institute of Innovation Research, Hitotsubashi University

Affiliated Fellow, National Institute of Science and Technology Policy,

MEXT

Ministry of Education, Culture, Sports, Science, and Technology

Ministry of Economy, Trade and Industry

MinistryofHealth,Labour, andWelfare

Ministry ofAgriculture, Forestry,andFisheries

Public R&D Budget in Japan

Basic44%

Applied27%

Investigation6%

Development23%

20 Billions US$


Comments from Commentators:

Evaluation results should be reflected to the budget on the uniform basis across different types of researches.

In active research fields, many researchers at different research units should compete to stimulate each other. It would produce the new discipline with high yield. So, the balance between the administrative efficiency and competition of researchers is very important.

Japanese evaluation system has no definite framework of ex-ante evaluation, so few projects/programs have clear targets to be accomplished.

An incentive system to R&D projects based on the results of evaluation should be carefully set in order to promote reasonable competition.



Session 2: Evaluation of Science and Technology Policy and Reflection of its Result in Japan

Current Situation and Challenges for Policy Level Evaluation in Japan

Dr. Jun SUZUKI

Professor, Technological strategy, S&T policy,Graduate School of Engineering Management,

Shibaura Institute of Technology

1. Backgrounds and track records for evaluation of science and technology policy

Case ： solar power generation Case ： HIMAC

Law on the Evaluation of Science and Technology Policy Conducted by the Government　　　　　　　　　　　　　　　　　　　（ Government Policy Evaluation Act, April 2004 ）

METI MEXT

METI Basic Plan for Policy Evaluations

Basic Plan for Policy Evaluations

achievement　・ ex-ante evaluation for 113 policy measures, an integration of 840 projects since FY2002　・ of which 75 policy measures(42 are R&D) subject to ex-post evaluation

achievement・ 3 evaluation procedures　 ex-ante, ex-post, and synthesized evaluation・ ex-ante and ex-post evaluation　 ex-ante: 78 projects ex-post: 8 projects・ achievement evaluation: 42 policy measures

He stressed the importance to employ the Balanced Score card and Behavioral Additionality to assess the effects of project performance.


It is very important to evaluate not only intended effects in the mid-term and long-term but also secondary effects.

Besides objectives and goals, management of the performance is also very important element of program management.

Project budget must be managed in a consistent manner throughout the project period.

It is unclear in Japan who is responsible for the evaluation results.

The evaluation system in EU sets importance in the balance of: accountability transparency justification learning trust in evaluation




Dr. Osamu NAKAMURA

Senior Researcher, Evaluation Department, AIST

Session 4: Evaluation of National Research Institutes

Revised Evaluation System to Reflect the FutureA model＠ National Institute of Advanced Industrial Science and Technology (AIST)

・ Independent administrative institution under METI・ To perform innovation-oriented basic researches

Sustainable Development by Full Research

AISTAIST SocietySocietyPublic resources

Outcomes

Full Research

People’sInterest(applause)

dream nightmare reality

Type-I basic research

Type-II basic research

Industrial development

AIST

Universities Industries

AIST places its highest priority on the pursuit of complete research, "Full Research", engaging Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research.

Outcomes

external reviewers:external reviewers:Specialists or IntellectualsSpecialists or Intellectuals

external reviewers:external reviewers:Specialists or IntellectualsSpecialists or Intellectuals

＊＊ Roadmap EvaluationRoadmap Evaluation ＊＊ Output EvaluationOutput Evaluation ＊＊ Management EvaluationManagement Evaluation

＊＊ Roadmap EvaluationRoadmap Evaluation ＊＊ Output EvaluationOutput Evaluation ＊＊ Management EvaluationManagement Evaluation

　　　　 Evaluation: Rating with commentsEvaluation: Rating with comments　　　　 Evaluation: Rating with commentsEvaluation: Rating with comments

Evaluation DepartmentEvaluation Department

Strategy of AISTStrategy of AISTStrategy of AISTStrategy of AIST

President of AISTPresident of AISTPresident of AISTPresident of AIST

Research UnitsResearch UnitsResearch UnitsResearch Units

internal reviewers:internal reviewers:Principal ReviewersPrincipal Reviewers

internal reviewers:internal reviewers:Principal ReviewersPrincipal Reviewers

Procedure of EvaluationProcedure of Evaluation

NationNation Evaluation CommitteeEvaluation CommitteeResource Allo

cation

Reorganization

Comments

＊ Roadmap for outcomes ＊ Output indexes ＊ Management　　

＊ Roadmap for outcomes ＊ Output indexes ＊ Management　　

Nation Evaluation Committee

to encourage researchers to perform Full Research

Evaluation from the view point of OUTCOMES

to know the level of performances and achievements

to make sure the research strategy 　

Notes: Evaluation is for Encouraging with Love. Evaluation is for Communication. Evaluation is for Strategy. Evaluation is for Creation. Evaluation is to be fed back. Evaluation reflects the Future. Evaluation is like a Compass for Voyage.

“Revised Evaluation System to Reflect the Future” (O. Nakamura, AIST)


Session 4: Evaluation of National Research Institutes

Evaluation System of RIKEN

Dr. Masayuki UCHIDA

Evaluation Section, Policy Planning DivisionRIKEN

・ Independent administrative institution under MEXT・ To perform comprehensive researches for natural science

RIKENAdvisoryCouncil

PresidentPresident

The Board ofThe Board ofExecutive DirectorsExecutive Directors

AC

AC

AC

RIKEN Frontier Research SystemDirector， FRS

RIKEN Brain Science InstituteDirector， BSI

RIKEN Center for Developmental BiologyDirector， CDBAC

RIKEN Genomic Sciences CenterDirector， GSCAC

RIKEN Plant Science CenterDirector， PSCAC

RIKEN SNP Research CenterDirector， SNP

Review Report Advice

RecommendationRecommendation

ACRIKEN Discovery Research Institute &Harima InstituteDirectors, DRI & HI

RIKEN Research Center for Allergy and ImmunologyDirector， RCAI

RIKEN BioResource CenterDirector， BRC

AC

AC

Response

2-1 Advisory Council System of RIKEN


It is very good to perform evaluation of a research institute based on its strategic roadmap, outputs, and research management.

If you want to improve the research environment, you should set a target and monitor indicators across years.

It must be emphasized to motivate researchers by setting appropriate criteria for evaluation.

Important elements to create a good business model of a research institute:1. Core competency2. Initiative and products (deliverables) from it3. Leadership based on a rational criteria4. Risk management (monetary risks, market risks, scientific risks,

non-technical risks, etc.)


Day2 Session 1: Applicability of logic model to policy evaluationDr.Gretchen Jordan (SNL)Mr.George Teather (NRC of Canada)

Session 2: Application of US PART for Japanese public RTD evaluationMr. Bill Valdez (DOE)Dr. Mike Holland (White House, OSTP)

Session 3: Application of EU FW6 Evaluation system for Japanese public RTDProf. Luke Georghiou (UM)Dr. Neville Reeve (EC)

Session 4: Current program evaluation methodsDr. Stephanie Shipp (NIST)

Session 5: Training program for policy evaluation/R&D evaluationProf. Nicholas Vonortas (GEU)Dr. Cheryl Oros (USDA)

Session 6: Newest evaluation toolsDr. Tom Fiddaman (Ventana Systems)


Summary:

We conducted extensive discussions on methodologies/ focuses on policy/program/project evaluations on Japanese science and technology policy system as a country case study with the guidance of WREN– for the first time ever.


Next Steps for Japan (METI):

We are trying to introduce logic models into planning and evaluation of METI projects and programs.

We consider that the some criteria in PART (US) or FP (EU), such as those for management, must be incorporated into Japanese S&T policy evaluation system.

We should get into the network of the world evaluation community in order to apply global top-level evaluation methodologies to Japanese S&T policy evaluation system.

Evaluation of Science and Technology Policy System in Japan:

Country Case Study

Thank you !


Appendix


R&D Budget of Japanese Ministries(including expenses for labors and facilities, 2005)

MEXT65%

METI17%

JDA4%

MHLW4%

MAFF3%

MLIT2%

others3%

MIC2%

36 Billions US$


3. Case study 2: Heavy Iron Medical Accelerator in Chiba (HIMAC)

(3) Heavy Iron Medical Accelerator in Chiba (HIMAC)

◆Background to development・　 Construction of Heavy Iron Medical Accelerator

in Chiba (HIMAC) began in 1984 as the world’s first heavy particle accelerator for medical treatment, based on the Comprehensive 10-Year Strategy for Cancer Control.

・　 Clinical experiments began in 1994.

・　 The device has been used in 2,184 people up to February 2005.

・　 HIMAC was provided to researchers both in Japan and overseas as equipment for utilization in biological experiments that would provide the basis for medical treatment, as well as in physical and engineering experiments. Over 500 outside researchers use the device each year.

・　 Highly-advanced medical treatment began in November 2003, and 341 patients have undergone treatment.

・　 R&D of downsizing the equipment was implemented to spread heavy particle cancer radiotherapy.

◆Special features ：・　 Causes no pain, placing less burden on patients

・　 Induces little or no side effects

・　 Can treat lung and liver cancer within a short period (less than 1 week)

・　 Can treat refractory cancers (e.g., that of the bone and soft tissues) that respond poorly to other treatment methods

Documents

Osamu NAKAMURA, Ph.D. Director for Technology Evaluation