Strategic Evaluation of Research

–Viewpoint of Outcome in the Evaluation of Research Units

in AIST–M. Daté, H. Tokunaga, S. Nakatsu, S. Ito,

H. Matsuhata, H. Ogi, A. Omori, S. Suto, M. Tajima, T. Tanaka, K. Urabe, O. Nakamura*, O. Nakamura, S. Kosaka, M. Takagi Sawada and N. Kobayashi

Natl. Inst. Adv. Ind. Sci. Technol. (AIST), JAPAN*Min. Econ. Trade Ind. (METI), JAPAN

1/15

Outline of AISTNew AIST (National Institute of Advanced Industrial Science and Technology) was established in 2001 as an Independent Administrative Institute.

ResearchersAdministrative StaffsTotal Employees

2,505704

3,209 As of April 2006

2/15

Mission of AISTBasic Principle is “Realization of a Sustainable Society”.

3/15

Research Fields in AIST

4/15

Organization of AIST

As of August 2006

5/15

Collaboration Dept

Auditor

President

Research Centers

Research Institutes

Research Initiatives

Senior Vice-PresidentVice-President

Emeritus Advisor

AIST Advisory Board

Research Coordinator

Emeritus Councilor

AIST Fellow

Research Units

adm

inis

trat

ive

un

its

Intellectual Property Dept

Technology Information Dept

International Affairs Dept

General Administration Dept

Human Resources Dept

Financial Affairs Dept

Research Facilities Dept

Intl Patent Organism Depositary

AIST Innov Center for Start-ups

Geo-information Center

Metrology Management Center

Tsukuba Adv Comp Center (TACC)

Planning HQs

General Administration HQs

Evaluation Dept

Safety & Environ Protection Dept

Public Relations Dept

Legal Office

Info Disclosure & Personal Info Protection Promotion Office

Gender Equality Office

Audit Office

Research Units of AIST

Research Centers (29)

Priority in investing research resources

Operated by top-down management

Pioneering and strategic projects within a certain period (<7 years)

Research Institutes (21)

Proposal of research subjects in a bottom-up manner

Mid- and long-term researches

Research Initiatives (7)

Response to sudden administrative needs

Aiming for new research centers or fields within a short period (<3 years)

As of August 2006

6/15

Evaluation of Research Units

Purposes of evaluation

2) Reference to the top management of AIST

3) Accountability to the Japanese Nation

1) Activation and efficient management of Research Units

1st Mid-Term (FY2001-2004) 2nd Mid-Term (FY2005-2009)

Evaluation based on Outputs and global level of research

Evaluation from the viewpointof Outcome

Viewpoint of evaluation

7/15

Definition of Terms in AISTOutputs:

Direct results in R&D (e.g., papers, patents, etc.)

・・ BudgetBudget・・ Human Human resourceresource・・ EquipmentEquipment

・・ Academic Academic paperpaper・・ PatentPatent・・ Geologic MapGeologic Map・・ StandardStandard・・ SoftwareSoftware・・ Process Develop.Process Develop.・・ DatabaseDatabase

・・ Popularization Popularization of productsof products・・ New research New research fieldfield・・ ServiceService・・ Saving energy Saving energy ・・ Risk Risk managementmanagement・・ DisasterDisaster preventionprevention

INPUT IMPACTOUTPUT OUTCOME

・・ GDP growthGDP growth・・ New employmentNew employment・・ InternationalInternational competition abilitcompetition abilityy・・ High QoLHigh QoL・・ Environmental Environmental protectionprotection・・ Security & safety Security & safety societysociety

Outcomes:Social and economical effects caused by outputs (e.g., products, standards, novel research fields, etc.)

8/15

Evaluation System in AIST

Outputs in the past Realized Outcomes

Expected Outcomesin the future

1) Roadmaps2) Outputs3) Management

PAST PRESENT FUTURE

Subjects of evaluation

2) Evaluation of Outputs toward Outcomes

3) Evaluation of Management toward Outcomes

1) Evaluation of Roadmaps toward Outcomes

9/15

Evaluation IndexesEvaluation of Roadmaps

Outcomes, Milestones, Technical Elements, Benchmarks

Evaluation of Outputs

Papers, Patents, Standards, Academic Activities, Press Releases, Prizes, Prototypical Products, etc.

Evaluation of Management

Research Strategy

Other Management

1) Full Research, 2) Innovations, 3) Human Resource Development

1) Overall Management, 2) Budget Management, 3) Risk Management

10/15

Reviewers in EvaluationExternal Reviewers (5-7 for each Research Unit)

e.g., academia, industry, journalism, etc.

Specialists and intellectuals outside of AIST

Internal Reviewers (3 for each Research Unit)

Principal Reviewers of AIST, etc.

Main Role: Evaluation of Roadmaps and Outputs

Main Role: Evaluation of Management

(Roadmaps & Outputs) : Management = 7 : 3

Allocation of grades

11/15

Evaluation Procedures

Resource, ReorganizationResource, Reorganization

Evaluation Committee

Evaluation Committee

1) Roadmaps2) Outputs3) Management

Evaluation DepartmentEvaluation Department

Activity Report

Presentation

Activity Report

PresentationComments on Evaluation

Claim of Reconsideration

Comments on Evaluation

Claim of Reconsideration

Grades Comments

Grades Comments

CorrectionCorrection

External ReviewersInternal Reviewers

External ReviewersInternal Reviewers

Research UnitResearch Unit

12/15

Evaluation Committeeof Ind. Admin. Inst.Evaluation Committeeof Ind. Admin. Inst.

PresidentPresidentJapanese NationJapanese Nation

Research Strategy of AISTResearch Strategy of AIST

Overall Evaluation Overall Evaluation

Survey of the Unit Directors (1)

Was the evaluation of this year useful?Was the evaluation of this year useful?

Was the evaluation worth all the efforts?Was the evaluation worth all the efforts?

FY2005

FY2005

FY

FY

13/15

Survey of the Unit Directors (2)

Was the viewpoint of outcomes useful?Was the viewpoint of outcomes useful?

Was the evaluation useful in improving management?Was the evaluation useful in improving management?

FY2005

FY2005

FY

FY

14/15

Comments of External ReviewersCATEGOLY PERCENTAGE

Affirmative comments on the evaluation system

20

Comments on the evaluation of roadmaps

14

Misunderstanding, insufficient explanation

14

Insufficient time or activity reports

12

Proposal to improve the evaluation system

10

Request for various evaluation indexes 9

Request for a longer interval of evaluation

8

Request to reduce the load of evaluation

6

Other comments 9

FY 2006

15/15