Embed Size (px)
DESCRIPTION
Reformation of National R&D Program Evaluation System. Woo Chul Chai. I. Overview of S&T in Korea. I. I. I. I. I. Contents. National R&D Program Evaluation System. Concluding Remarks. 2. Overview of S&T in KOREA. 3. Changes of S&T Environments in Korea. 1960s. 1970s. 1980s. - PowerPoint PPT Presentation
Citation preview
Woo Chul Chai
Reformation of National R&D ProgramEvaluation System
Overview of S&T in KoreaI
National R&D Program Evaluation System
II
Concluding RemarksIII
2
Contents
3
Overview of S&T in KOREA
1960s 1970s 1980s 1990s 2000s
Primary goodsPrimary goods
Light industry goods
Light industry goods
Light & heavy industry goods
Light & heavy industry goods
Heavy industry goods & electronic products
Heavy industry goods & electronic products
Electronic & transport products
Electronic & transport products
GRIs established(e.g. KIST)MOST established in 1967Daedeok Science Town built in 1974
Building R&DinfrastructureBuilding R&Dinfrastructure Promoting
R&DPromoting R&D
Enforcing the national R&D programs
Promoting the university-based researches (e.g. SRC, ERC)
Enhancing technology innovationEnhancing technology innovation
Increasing efficiency of R&D investment (coordination of S&T- related policies)Planning Total Roadmap
Industry- oriented policy
Technology- oriented policy
S&T policy
direction
Change in focusing industry
Policy trend
Changes of S&T Environments in Korea
4
Brief History of National R&D Programs
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
00
01
02
08
Year
National R&D ProgramNational R&D ProgramMOST
Industrial Technology ProgramIndustrial Technology ProgramMOCIE
Information & Communication Technology ProgramInformation & Communication Technology ProgramMIC
Health & Medical Technology ProgramHealth & Medical Technology ProgramMOHW
Environmental Technology ProgramEnvironmental Technology ProgramME
Agricultural Technology ProgramAgricultural Technology ProgramMAF
Construction & Transportation Technology ProgramConstruction & Transportation Technology ProgramMOCT
Academic Research Promotion ProgramAcademic Research Promotion ProgramMOE
MEST
MKE
New Growth Engine ProgramNew Growth Engine Program
’08
5
S&T Development in Korea (Quantitative Growth)
Science Competitiveness Technology competitiveness
0
5
10
15
20
25
302001 2002 2003 2004 2005 2006
14th12th
16th
19th
15th12th
21th17th
27th
8th
2th
6th
Worl
d r
an
ks
Source : IMD reports
2007 2008
6th
14th7th
5th
6
7
National R&D Program Evaluation System
History of National R&D Program Evaluation
Increasing Needs of Effectiveness of R&D Investment
Survey·Analysis·
Evaluation
Survey·Analysis·
Evaluation
Enhancement of
Effectiveness,Efficiency &
Accountability
Enhancement of
Effectiveness,Efficiency &
Accountability
○ ’97~’05
○ S&T Framework Law Enacted (’01.1)
○ Focus : Appropriateness of Input and Output
○ ’06~’07
○ Law on Performance Evaluation and Performance Management of National R&D Program Enacted (‘05.12)
○ Focus : Performance Achievement based on the Results of R&D Activities
○ ’08 ~
○ Law on Performance Evaluation and Performance Management of National R&D Program Revised (‘08.2)
○ Focus : Performance Evaluation & Management for Enhancing Effectiveness of R&D Investment
Entering Stage
Performance EvaluationPerformance Evaluation
Growing Stage
Performance
Management&
Evaluation
Performance
Management&
EvaluationTransformational
Stage
Effectiveness of R&D
Investment
8
Architecture of Program Evaluation in NES
MOSF
Ministries
EvaluationGuideline
Evaluation Results
To evaluate the appropriateness of Self-Evaluation
To review evaluation procedure and methods of Self-Evaluation
Annually practiced by Ministries
Implementation based on self-made performance indicators and methods
In-depth evaluation on major national R&D programs
Long-term/large-scale programs
Joint program among ministries
Programs which need to remove redundancy and require connectivity
Programs at national issue
Mainly examined by the Evaluation Committee
Specific Evaluation
Self Evaluation
Planning Self-Evaluation according to MOSF’s guideline
Implementing Self-Evaluation
Basic Plan for Performance Evaluation (5 yr)
Action Plan for R&D Evaluation (annually)
MetaEvaluation
NSTCReview of Plan Reports of Results
9
Lifecycle of National R&D Program Evaluation
Feasib
ilityA
naly
sis
Sp
ecifi
c (In-D
epth
) Eval
Follo
w-u
p E
val
Self-Eval
Meta-Eval
Ex-ante Ex-PostImplementation
Time
Evaluation Range
Startyear
+ 1year +3year +3year
Self-Eval
Meta-Eval
Self-Eval
Meta-Eval
Closing
year
10
Self/Meta-Evaluation of National R&D Programs (1)
Designed after PART (Program Assessment Rating Tool) of the U.S. Federal Government
Line Ministries assess their own programs every 3 years
The assessment is based on 25 checklist (questions) for all types of programs
- Types of programs: R&D (Basic, Applied Research, Development), Infrastructure Investment, Procurement of Large-scale Facilities and Equipment, Human Resource Development
MOSF reviews the assessment results and reflects them in annual draft budgets and the National Fiscal Management Plan
11
Self/Meta-Evaluation of National R&D Programs (2)
Answers to the questions take the form of “Yes (5)” or “No (0)
- In case of the questions regarding the achievement of
program goals, 2-scale answers (5, 0) is given.
A different score is assigned to each question and the result of
assessment is given by the sum of score
- Classified as “Effective (95-100),” “Moderately Effective (90-94),”Adequate (75-89),” and “Not Effective (0-74).”
12
Evaluation Items and Score (3)
Section Evaluation Items Evaluation Point Score
Planning
1. Objectiveness and
Feasibility
- Clarity of Goals
- Feasibility of Program Contents5
2. Rationality of Program
System
- Appropriateness of Budget Supply
- Appropriateness of Program Delivery
System
- Role between Program Players and
Appropriateness of Cooperation
System
5
Execution
3. Appropriateness of
Program
Management and Execution
- Appropriateness of Budget Execution
- Appropriateness of Program Schedule 5
4. Appropriateness of
Performance
Management
- Appropriateness of Strategy and Plan
for
Achievement of Performance
- Level of Performance Management
5
Result
(Performanc
e)
5. Achievement of
Performance Standard and Indicator (5)
- Key Indicator (3)
- General Indicator (2)65
Utilization 6. Utilization of Evaluation
Result -Implementation of Feedback 1513
14
Concluding Remarks
Limitations
Little enthusiasm from line ministries
Line ministries did not set up a clear framework of mission and
strategy
Performance indicators were not derived from ministerial missions
in a systematic way
Assessment of performance relies on subjective assessment by
outside experts and in-house staff, not systematically utilizing
indicators
Cooperation and coordination among players are not sufficiently
made
15
Future Works for Evaluation
Developing Strategic Performance Management Frame
- Reviewing Performance Indicators, Monitoring R&D Activities,
etc.
Enhancing Education & Consulting for R&D Program
Officers
- Organizing Education Program of Performance Management
- Developing Logic Model Manual for Various R&D Program Types
Linking Evaluation Result with Budget Allocation16
