Models of Evaluation For Research

Proposals in Turkey

Prof. Dr. Omer CEBECIVice President – Funding, TUBITAK

(The Scientific and Technical Research Council of Turkey)

&S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU,

F. COSKUN, O. OZPEYNIRCI, B. DIKMEN,G. KOZANOGLU, H. GULER & O. KARA

SUMMARY

• Research funding in Turkey, TUBITAK & recent boost

• Three-dimensional evalution used by TUBITAK for the evaluation/selection of research project proposals grouped under three categories:

• (1) curiosity driven academic research, and• (2) customer driven applied research,• both in universities and research institutions, and• (3) technological and innovation driven research

conducted by the private industry, plus• (4) research equipment & infrastructure.

• Details of the “Phrase-anchored rating scale”

Actors of Turkish National Science and Technology System (policy makers)

BTYK- Supreme Council of Science and Technology

ME-Ministry of Education

MIT-Ministry of Industry and Trade

TUBITAK-The Scientific and Technical Research Council of Turkey

HEC-Higher Education Council (YÖK)The Turkish Academy of Sciences (TÜBA)

KOSGEB- Small and Medium Sized Industry and

Development Organizations

TAEK- The Turkish Atomic Energy Commission

TOBB-Union of Chambers and Commodity Exchange of Turkey

DTM-Foreign Trade and Treasury

TTGV- Technology Development Foundation of Turkey

DPT- State Planning Organization

TÜRKAK- Accreditation Board

TUBITAK National Research Centers and Institutes

Marmara Research Center (MAM)Institute of Energy

Chemistry and Environment InstituteFood Institute

Materials Institute Information Technologies Research InstituteEarth and Marine Sciences Research InstituteInformation Technologies and Electronics

Research Institute (BİLTEN)National Electronics and Criptology

Research Institute (UEKAE)Defence Industry Research and Development

Institute(SAGE)Basic Science Research Institute

National Academic Network CenterGenetic Engineering

and Biotechnology Research InstituteNational Metrology Institute

Turkish Industry Management InstituteNational Observatory

Actors of Turkish National Science and Technology System (science and research performers)

Turkish State

Universities

Turkish Industry

OtherNational Research

Centers and Institutes

Turkish Private

Universitiesand Research

Centers

President of Republic

Prime MinisterBTYK

YÖK TÜBİTAK

STB

BTYK- Supreme Council of Science and TechnologySTB- Minister of Industry and TradeMEB- Ministry of National EducationTÜBİTAK- The Scientific and Technological Research Council of TurkeyYÖK- Higher Education Council (YÖK)TÜBA- The Turkish Academy of Sciences (TÜBA)DPT- State Planning OrganizationTÜRKAK- Accreditation Board

MEB

Universities DPT TÜBA

TÜRKAK

TPEKOSGEB

TSE

TTGV

R&D InstitutionsTPE- Turkish Patent InstituteTSE- Turkish Standards InstitutionDIE- State Institute of Statistics DTM- Undersecretariat of the Prime Minister for Foreign Trade TOBB-Union of Chambers and Commodity Exchange of TurkeyKOSGEB- Small and Medium Industry Development OrganizationsTTGV- Technology Development Foundation of Turkey

DTM

TOBB

DIE

Turkish National Science, Technology and Innovation System

National S&T Indicators

EU-15 Values (2000-2001):Number of triadic patents per million population:36

Number of papers per million population:822

GERD per capita population:467. 6 (PPP $)GERD as a percentage of GDP:1,89

Total R&D personnel per 1000 employment: 10.4

0

0,5

1

1,5

2

2,5Number of triadic patents per million population

Number of papers per millionpopulation

GERD per capita population

GERD as a percentage of GDP

Total R&D personnel per1000 employment

EU-15

Japan

US

Germany

0,5

1

GERD as a percentage of GDP

Spain

EU-15

Hungary

Turkey

National S&T Indicators

Total R&D personnel per1000 employment

Number of triadic patents per million population

Number of papers per millionpopulation

GERD per capita population

Scientific Publications

0,460,52

0,57

0,64

0,86

0,94

1,11

0,40

0,50

0,60

0,70

0,80

0,90

1,00

1,10

1,20

1998 1999 2000 2001 2002 2003 2004

Year

Sh

are

of

Tu

rkey

in W

orl

d P

ub

licat

ion

s (%

)

Knowledge as a Tool

Knowledge is a tool that can be used for a variety of social objectives, including:

Meeting Basic Human Needs Increasing Safety & Security Improving the Quality of LifeEconomic Growth and Development

100.000 $Communication Satellite

4.000.000 $Microprocessor (Si) Chip

10.000 $Combat aircraft

3.000 $Fiberoptical Cable

100 – 1.000 $Passenger Aircraft

10 – 100 $Automobile

1,5 $Aluminium

50 centSteel

5 centCement

1 centConcrete

Knowledge - R&D Cost per Kg

8

10

15

11

10

8 8 7

5

8

10 9

11 10

9 8

7

8 8 9

12 13 13

9 8 8 8 8 9

10 10 10

16 17

46

0

5

10

15

20

25

30

35

40

45

50

1971 1973 1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005

ON

BİN

DE

Share of TUBITAK inTR Fiscal Budget

Research Proposals

*from universities and research institutions:1) curiosity driven academic research2) customer driven applied research

*from private industry:3) technological & innovation driven

research

TUBITAK Research Project Proposals

Science and Technology Project Proposals

0

500

1000

1500

2000

2500

3000

1999 2000 2001 2002 2003 2004

Years

Nu

mb

er o

f P

rop

osa

l

Career

International

Sector

Units

Infrastructure

Research

Running Research Projects

264

430

587694 750

885 889 929 899833 837

982

1212 11991353

2600

0

500

1000

1500

2000

2500

3000

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

EVALUATION & MEASUREMENT

TARİHTEN BUGÜNETARİHTEN BUGÜNESAĞLIK ve SAĞLIK ve ÖLÇMEÖLÇME

• Ateş ve nabız – 1625(Santario ve Galileo)

• Torricelli - 1643• Tansiyon - 1733 (Hales)

• 1896 - Riva-Rocci• Korotkov - 1905• Ateş - nabız - tansiyon ölçüm ve

kayıtları ile hasta izleme - 1920

ÖLÇME ve DEĞERLENDİRMEÖLÇME ve DEĞERLENDİRME“ Ölçemediğinizi anlıyamazsınız ! ”

Lord Kelvin (1824 - 1907)

(If you can’t measure it -you don’t understand it)

“Her hesaba katılması gerekeninsayılır olması gerekmez ! “

Albert Einstein (1879 - 1955)(Not everything that can be counted counts &

not everything that counts can be counted)

Çıktı mı ? Sonuç mu ? Etki mi?

Çıktıların sonucu olarak gerçekleşen ETKİ !

OUTPUT – OUTCOME – IMPACT

IMPACTS achieved as consequences of the

OUTCOMES resulting from the

realisation of the OUTPUTS

OUTPUT – OUTCOME – IMPACT

IMPACTS achieved as consequences of the

OUTCOMES resulting from the

realisation of the OUTPUTS

( My eye jelly project )

Intended / Expected / Desired IMPACT :

. . . in full control of my mandate . . .

Unintended / Unexpected / Unwanted

BUT ACTUAL REAL IMPACT :

. . . under strict control of my wife ! . .

Evaluation Criteria

three-dimensionalevaluation criteria

developed in collaboration withresearchers & reviewers

1) Curiosity DrivenAcademic Research

- the three dimensions -

1. intellectual/scientific/professional merit

2. expected impact of the anticipated outcome

3. achievability of the research with the proposed research team, equipment/facilities and methods

1. research and development merit

2. expected impact of the anticipated outcome

3. achievability of the research with the proposed research team, equipment/facilities and methods

2) Customer Driven Applied Research

- the three dimensions -

1. technological level of the research

2. innovative level of the product/outcome

3. feasibility of the process

3) Technological & Innovation

Driven Research- the three dimensions -

Evaluation Process

all three-criteria are given equal weights

Phrase-Anchored Rating Scaleversus

Likert Scale

sub-criteria phrases describing

*very competitive *competitive *not-competitive

features of the proposal

Very competitive

*scientifically and professionally outstanding & very well justified project

&*points to an opportunity for a major

contribution to the advancement of the knowledge and/or to the resolution of a problem of practical importance

Competitive

*scientifically and professionally competent and justified proposal which will make a contribution to the advancement of knowledge &/or the resolution of a problem of practical value

& therefore*support is suggested if funds are

available

Not competitive

*work routine in character*scientifically and professionally

unsatisfactory and poorly organized

Evaluation Process

evaluate 10-15 proposals by referring to the

sub-criteria phrases

a meeting for a final verdicta meeting for a final verdict

Panels

5-8 Individual Reviewers

Curiosity / Merit = V. Comp.

• Outcomes have high potentials for publication in journals or books listed in international indexes

• The originality of the work has been supported by extensive and critical literature survey

• Hypothesis for evaluating the research topic is very well defined

• Explanation and analysis of the expected outcomes reveal the superiority of the work in comparison to the existing science and technology.

Curiosity / Merit = Not comp.

• Scientific consistency and the rationale of the research are not clearly explained.

• A clear scientific / technological question is not put forward.

• Project is more like an investigation / data collection / routine work.

• References provided include similar studies & literature survey does not point out the basis / importance of the project.

• The research is based on unreliable data and hypothesis.

Curiosity / Impact = V. Comp.

• Very important in terms of sustainable development of the country /very likely to be implemented to find solutions for the problems of society

• Very likely to be employed in different scientific & technological fields

• Very likely to generate new projects• Commercialization potential of the

outcomes is very high• The project is supported by international,

national or industrial sources.

Curiosity / Impact = Not comp.

• Potential for adding value to science and technology is low

• Subject of the project is not among the priorities of the country

• Not likely to result in intellectual property worth-protecting.

Curiosity / Achiev. = V. Comp.Project Team• The team is experienced in national /

international projects related with same/similar subject

• They have experience as advisors /authors / referees / editors / book authors

• The can allocate enough time for the project• Competencies / responsibilities / roles of the

team members are well defined and adequate

• End users of the project outcomes are also members of the team.

Curiosity / Achiev. = Not Comp.Project Team• The team is not experienced in conducting

projects of this size• The knowledge and awareness of the team

is not sufficient• They do not have important publications in

the subject of the project• Some of the team members are irrelevant

for the project• Essential competencies are lacking

Curiosity / Achieve. = V. Comp.

Infrastructure• Infrastructure of the institution is very

adequate for the project• Additional equipment requested within the

scope of the project is very compatible with the existing infrastructure and the project

• Existing sources / equipment are used rather than purchasing new sources

• Requested equipment can also be used in other / future projects

Curiosity / Achieve. = Not Comp.

Infrastructure• Infrastructure of the institution is not

adequate for the project, unless supported with major equipment

• Equipment requested within the scope of the project is not compatible with the existing infrastructure and the project

Curiosity / Achieve. = V. Comp.

Methodology• Approach / methodology are very well

designed to reach the target• Methodology is correct and well-defined;

and standard methods and literature are cited

• Preliminary experiments have been conducted to rationalize the hypothesis

• Alternatives (plan B) have been considered if difficulties are encountered

Curiosity / Achieve. = Not Comp.

Methodology• Methodology is not adequate to reach the

target• Relations between the experiments and

hypothesis are not well defined• Methodology is not explained with a

common scientific basis• Possible problems and limitations are not

considered• Statistical analytical requirements are not

considered

Curiosity / Achieve. = V. Comp.

Timeline• Proposed period & time schedule are

realistic

Budget• Proposed budget is realistic and well-

justified• Project is also supported by other

institutions

Curiosity / Achieve. = Not Comp.

Timeline• Proposed period and time schedule are not

synchronized• Time schedule is not adequate

Budget• Budget is not well-define and requested

amount is too low / high• There is no possibility of support from

other institutions

Customer / Merit = V. Comp.• Aims to develop a national / international novel

technology (methodology, system, product, process / technique)

• Brings comparable superiority to the existing system• Outcome will be a technology to be protected under

the intellectual property rights• Work consists of a scientific and rationale approach• An interdisciplinary project with the collaboration of

the Public- Private Industry-University-Research Institutions

Customer / Merit = Not comp.

• Literature survey and market survey are not satisfactory, work is based on unreliable data

• No scientific / technological rationale and integrity in the project

• It is a study that had been done before in the country

• Aim, objective and motivation are not clear• Outcome is not qualified as

“applicable/usable”• Not related with a real need / problem

Technological Level = V. Comp.

• Technology / product developed aims to fill a gap in the existing technology or replace the existing technology within the following 3 years

• Has an interdisciplinary approach to solve more than one problem

• Rationale of the R&D is well established (theoretical / analytical / experimental)

• Added value of the anticipated outcome of the R&D project is considered

• Has contribution to increase R&D staff• A doctorate / masters study is incorporated within

the project and the outcomes have potentials to be published in national/international journals.

Technological Level = Not comp.

• Literature survey and market survey are not adequate

• No scientific/technological advance and integrity in the project

• R&D rationale (analytical and /or experimental) is not adequate

• Aim, objective and motivation are not clear.- Procedure that has been used will not provide a change / improvement in the methodology and technology

Innovation Level = V. comp.

• Outcomes may lead to spin-off company• Outcomes include production standards

and technical specifications• Outcomes will affect other sectors as well• Outcomes very likely to increase the

export capacity of the country• Very likely to be a basis for generating

new projects / products

Innovation Level = Not comp.

• Outcome is not qualified as “applicable/usable”

• Period of usefulness is very limited or not effective

• Does not provide new areas of work and as a result does not increase employment

• Expenditure for the R&D study is far beyond the expected economical benefit of the product

• Support provided by the private organization is not sufficient.

CommentsReviewers (hundreds):• The model reduced the,

– burden of the review process on them, &– subjectivity and variability of the opinions of the

individuals• Convenience of selecting from a comprehensive

list of phrases as well as • the freedom of offering their own judgments

Researchers (thousands):• Sub-criteria phrases to be very instrumental in

guiding the development of their proposals

Thank you

Hundreds of scientists, researchers and reviewers participated in

workshops for the development of the criteria and phrases listed in the

tables. Their contributions are gratefully

acknowledged.