Embed Size (px)
Citation preview
Jan Dvořák, Jiří Souček:
The R&D Efficiency Evaluation in the Czech Republic
CRIS2008 – Maribor, Slovenia – June 2008
InfoScience Praha, s.r.o.
2
Agenda
The Czech R&D IS (JD)
The Efficiency Evaluation in the CR (JD)
Efficiency Evaluation in general (JS)
3
The R&D Information System of the Czech Republic A CRIS covering the R&D subsidized from the State Budget
of the Czech Republic Run by the Czech R&D Council Contains data on
Funding providers, Programmes, Calls R&D Projects (since 1994, ~30,000) Institutional Research Plans (since 1999, ~880) R&D Results – publications, patents, technologies (since 1998,
~530,000 records, ~420,000 unique results)
Data from funding providers
4
http://aplikace.isvav.cvut.cz
5
The R&D Efficiency Evaluation
The setup The R&D IS contains all results of all research that has been
supported from public sources in the CR The State is the predominant supporter of basic research in the CR
The „investor's“ point of view Results per unit funding
Runs in 2005, 2006, 2007
6
Results → Points
Results are assigned points Example from the 2007 Evaluation
Article in an impacted journal → 5 + 15 x IF / median( IF ) over disciplineusing data from ISI JCR
Article in a non-impacted, reviewed periodical → 1 (Czech & Slovak)2 (other languages)
Monograph → 12.5 (Czech & Slovak)25 (other languages)
Proceedings contribution → 0.1 (Czech & Slovak)0.2 (other languages)
7
Results → Points
Example from the 2007 Evaluation (cont'd): Patent → 500 (EPO, USTPO, JPO)
50 (other issuer, incl. the CzechIndustrial Property Office)
Verified technology, breed → 75 Prototype, software → 25
Special handling of Social Sciences and Humanities
The point-assignment rules are agreed upon by the R&D Council members
8
Results → Points
The point assignment has been evolving E.g. monograph:
2005 2006 2007
Czech & Slovak 5 5 12.5
other language 10 20 25
9
Point distribution
R esults
R e se a rcha ctiv ity
Institu tionsvia research activities
(until 2007)
Institu tionsvia results
(since 2008)
Program m e
F undingprovider
Points
Points
Cost
PointsCost
Points Cost
Points Cost
10
Point distribution
M:N relationships need a distribution factor
result : research activity → result points divided uniformly
research activity : institution → result points divided in the proportion of State Budget funding
11
Index
Points( X ) / Cost( X ) → Index( X ) Institution Programme Funding provider Whole R&D
Index( X ) / Index( Whole R&D ) → Relative Index( X )
12
Relevant data selection
Sliding window: Evaluation in year Y:Research activities ending in [ Y-5, Y-1 ]
Infrastructure projects excluded
13
Sample
17,969 1821 235 516The whole Czech state (co)funded R&D totals
27,920 221564 040The Czech public universities sector totals101%83%78%28,11 52442 790Institute of Chemical Technology Prague80%87%69%22,31 67637 318Brno University of Technology
112%133%139%31,31 96561 476Masaryk University, Brno56%80%69%15,62 93345 750Czech Technical University in Prague
130%112%125%36,35 923214 847Charles University in Prague
Eval. 2007
Eval. 2006
Eval. 2005
IndexState funding [MCZK]
Result poin
ts
Relative index (within the public universities sector)
Evaluation 2007University name
14
Sample
0,00%
20,00%
40,00%
60,00%
80,00%
100,00%
120,00%
140,00%
160,00%
CharlesUniversity in
Prague
MasarykUniversity,
Brno
PalackýUniversity in
Olomouc
Institute ofChemical
TechnologyPrague
BrnoUniversity ofTechnology
CzechTechnical
University inPrague
2005
2006
2007
15
Points by result type
counts pointsarticle in impacted journal 14,7% 65,5%article in reviewed periodical 23,4% 5,8%book 3,5% 9,7%book chapter 5,2% 1,4%proceedings contribution 51,5% 1,3%patent 562 0,3% 3,8%technology, breed 0,8% 10,2%prototype, software 0,5% 2,2%
29 773 809 75847 445 72 1717 164 119 767
10 592 17 765104 340 16 639
46 7881 676 125 7001 077 26 929
202 630 1 235 516
16
Changes in 2008
Modified point-assignment rules
Esp. for articles in impacted journals
Institutions evaluated by results
Institution gets points for a result in the proportion of the authors affiliated with it to the total number of authors
Part of the “Reform of the system of research funding by the State”
17
General principles of the Evaluation
Systematic part (the logic of R&D)
Results evaluated by the point-system Dividing points to individual projects Summing up points to Institutions Calculation of indexes and relative indexes
Specific parameterization for points-systems
Depends on the type of the result Using impact factor of the journal Using properties / attributes of the result
Needed data on objects in R&D
Projects, institutions, results Links projects – results Links institutions – results
18
Principles or the impersonal (objective) evaluation
Is determined by rules (not by opinions) Impartiality - does not depend on opinions Non-refutability – depends on hard facts Is possibly imperfect, but the inaccuracy is random and suppressed
by the aggregation (25 results 5 times better precision) Systematic inaccuracies can be traced and corrected
Basic characteristics Non-refutability Impartiality
19
The importance of the impersonal (objective) evaluation
Serves not only for R&D Council, but
For all workers in R&D field They can independently judge the contribution of others
(projects, teams, institutions, …)
Expected reception of a published paper is approximately equal to the impact factor of the journal (it is not necessary to wait many years for citations to accumulate)
Impersonal evaluation gain more weight than the peer review, since it is non-refutable
20
Needs of ERA (European Research Area)
In the R&D information infrastructure & evaluation Accessibility of the information on projects, funds, calls, results
(also on the national level) Accessibility on the data supporting the evaluation
The way : Integration of data from national CRISes (just in progress) Building evaluation data system
Efficiency of the allocation of resources In economics – market forces In research area - there are, in principle no market forces
The role of market forces should be substituted by the impersonal (objective) evaluation.
21
Our vision
The creation of the consortium focused to evaluation of R&D results (in the 7 FP – the infrastructure of the research)
WE ARE LOOKING FOR PARTNERS AND COOPERATION!
If you are interested in R&D evaluation please contact us!
22
Thank you for your attention
Jan Dvořák
Jiří Souček
Martin Souček
Richard Papík
