View
222
Download
2
Category
Tags:
Preview:
Citation preview
Citation analysis in research evaluation
Wouter Gerritsma, Marianne RenkemaInformation Specialists, Wageningen UR Library
Introduction
Introduction Citation Indexes Citation analysis of scientific journals
Excercises Citation analysis of researchers and groups
of researchers
When research evaluation/citation analysis? Research assement of graduate schools (5
year cycle, stipulated by the VSNU) Other research groups are reviewed in a similar
way Benchmarking becomes more important
Grant applications Job positions
Sooner of later, you will be subjected
to such an analysis
Objective
How are citation analyses performed, and what are their shortcomings or weaknesses?
What can you do to enhance your results
References and citations what is the difference?
If paper R contains a bibliographic footnote using and describing Paper C, then R contains a reference to C, or R cites C, and C has a citation from R or is cited by R.
C is a (cited) reference contained in R. R is the source of the reference.
Source: Moed (2005) based on Price (1970)
Citation Indexes
Institute for Scientific Information Founder Eugene Garfield Part of the Thomson group (commercial publisher)
Index articles of about 7800 journals, standard bibliographic information + reference lists
Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI)
They have monopoly position for citation data; they are the Standard
CC, WoS, JCR, ESI
CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data)
WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC)
JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals.
ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines.
Database Coverage WoS/JCR/CC
Excellent Good Moderate
Molecular biology & Biochemistry
Applied physics & Chemistry
Other Social Sciences
Biological Sciences related to humans
Biological Sciences related to plants & animals
Humanities
Clinical medicine Psychology & psychiatry
Physics & Astronomy Other social sciences related to medicine & health
Mathematics / Engineering / Economics
Source: Moed (2005)
Are there competitors ?
Scopus (citation tools are being tested) Google Scholar (http://scholar.google.com) PsychInfo/SciFinder (article indexes in Desktop
library) ArXiv (Physics) Spires (high energy physics) Citeseer (ICT) OA Initiatives
Citation analysis of journals
Warning !
Often it has been attempted to use Journal Impact factors in some way to judge the performance of a group or individual scientist.
This is proven very bad practice!
Journal citation analysis
What are impact factors Are there other journal performance
measures? What are the pitfalls of the IF How can I make use of this knowledge?
Journal Citation Reports
Report 3 measures Impact factor Immediacy Index Cited half life
0 2 4 6 8 10
Years after publication
Cita
tions
/yea
rCited half-life
50% citations 50% citations
31
Immediacy index Window
Impact Factor Window
Definition of Impact factor
The impact factor of a journal J in year T is defined as follows:
“The number of citations received in year T by all documents published in J in the years T-1 and T-2”
÷“The number of citable documents published in J in
the years T-1 and T-2”
Citable items
Cited items (nominator) include: Research articles, review arrrticles, notes,
letters, editorials, news items, corrections and meeting abstracts
Citable items (by definition) Articles, notes and reviews
Lancet (2002)
Type of document
No. Docs Cites Cites/Doc
Articles/reviews (citable items)
1,544 (a) 13,106 8.5
Other types 4,899 2,564 0.5
Total 6,443 15,670(b) 2.4
JCR Like IF = (b)/(a) = 10.2
Impact factor
Performance measure for journals“… it is also used for assessment of the quality of
individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997)
Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33(1): 1-7. http://dx.doi.org/10.1016/S0008-6363(96)00215-5
50 % of articles generate 90% of all cites
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314(7079): 497-502. http://bmj.bmjjournals.com/cgi/content/full/314/7079/497
Some other points
IF measures citation impact of articles in 2nd & 3rd year after publication. Therefore biased towards journals revealing rapid maturing.
Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can therefore not be compared.
Life: JCR & ESI
When do you make use of IF
Answering the questions where do you want to publish your manuscript.
Other questions, to be asked Reach colleagues ? In which article indexes included? Open access?
Citation analysis of paper, scientist or group
Science Citation Indexes
Web of Science is comprised of three Citation Indexes
Science Citation Index Exanded Social Science Citation Index (SSCI) Arts & Humanities Citation Index (AHCI)
Searching WoS
General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses Not all citations are found due to errors in citing,
indexing etc. Cited reference search: searches in the
references of source data Not all articles are found. Non cited articles are
missing
Author searches
Use the index !
But what does it mean?
Publication A (2003), 25 citations (Mol. Bio.) Publication B (1996), 17 citations (Math)
Essential Science Indicators (ESI)
Analytical database of SCI, covering 10 years + current year building
Comparison between Countries, Institutes, Scientists and Journals
Hot papers / Highly cited papers Research fronts Baselines
Publication age : world averages citations (1)Citaties in verchillende onderzoeksvelden
0,00
50,00
100,00
150,00
200,00
250,00
300,00
350,00
400,00
450,00
199319951997199920012003
Publicatie jaar
Cit
ati
es
Agricultural Sciences
Molecular Biology &Genetics
1.00 %
10.00 %
1.00 %
10.00 %
Publication age : world averages citations (2)Citaties in verchillende onderzoeksvelden
0,10
1,00
10,00
100,00
1000,00
199319951997199920012003
Publicatie jaar
Cit
ati
es
Agricultural Sciences
Molecular Biology &Genetics
1.00 %
10.00 %
1.00 %
10.00 %
In practiceAgricultural sciences
0,1
1
10
100
199819992000200120022003
Publication year
Cit
atio
ns
1.00 %
10.00 %
AVG
Group 1
Group 2
Group 3
In practice (WIAS, plant & animal sciences)
0,1
1
10
100
1000
1994 1996 1998 2000 2002 2004
Publication year
Cit
atio
ns Citations
Average
Best 10%
Best 1%
Relative impact of a research institute
Subfield All groups Group 1 Group 2 Group 3
Agricultural Sciences
3,82 3,86 3,87 3,60
Biology & biochemistry
0,91 1,55 0,44 1,09
Chemistry 1,76 1,76
Clinical medicine 1,73 1,81 1,11
Microbiology 1,70 0.57 1,73
All subfields 2,06 2,08 2,26 1,84
Wageningen in Dutch perspective (2004 data)Institute
No. highly cited papers Papers Citations
Cites/ paper
World Rank
Univ Utrecht 343 23.938 286.385 11,96 68
Univ Amsterdam 338 20.730 262.183 12,65 75
Leiden Univ 280 18.565 262.183 14,11 77
Vrije Univ Amsterdam 248 15.353 200.385 13,05 111
Wageningen UR 230 14.707 151.227 10,28 171
Erasmus Univ Rotterdam 217 13.450 208.290 15,49 103
Univ Groningen 217 15.800 158.830 11,76 128
Univ Nijmegen 152 14.700 172.604 11,74 140
Delft Univ Technol 145 10.105 70.204 6,95 362
Univ Maastricht 112 8.636 95.775 11,09 276
Eindhoven Univ Technol 92 6.199 43.567 7,03 522
Univ Twente 75 5.679 39.144 6,89 562
Tilburg Univ 11 1.818 7.175 3,95 1714
Strength & Weaknesses of Wageningen UR
Subfield
Highly cited papers
Papers CitesCites/Paper
World rank (citations)
ENVIRONMENT/ECOLOGY 40 1878 20187 10,75 3
AGRICULTURAL SCIENCES 39 2212 14502 6,56 3
PLANT & ANIMAL SCIENCE 99 4249 39280 9,24 5
MICROBIOLOGY 8 951 15239 16,02 24
PHARMACOLOGY & TOXICOLOGY 2 189 2946 15,59 170
BIOLOGY & BIOCHEMISTRY 2 1421 15408 10,84 197
GEOSCIENCES 7 299 3051 10,20 222
MOLECULAR BIOLOGY & GENETICS 0 467 8237 17,64 263
CHEMISTRY 8 995 9616 9,66 284
SOCIAL SCIENCES, GENERAL 3 203 692 3,41 352
CLINICAL MEDICINE 14 791 15563 19,68 404
ENGINEERING 3 269 943 3,51 601
Example of selection of candidates for a professor in 2004
Candidate# Papers 1994-2003
#Citaties Relatieve Impact
RI 1994-1998
RI 1999-2003
#papers bij top 10%
#papers bij top 1%
a 80 1565 1,64 1,76 1,52 4 2
b 65 498 1,93 1,84 1,95 17 1
c 93 972 1,15 1,39 0,9 8 0
d 88 1886 1,86 1,69 1,94 16 3
e 57 346 0,75 0,58 0,83 3 0
Life WoS & ESIExercises WOS & ESI
What questions should you ask
Database coverage Accuracy of the counts Time period of the exercise Self citations Normalization? Verify the collected data
Winding down
What is in a name? Persons (female PhD students!, initials!) Groups (Chair groups, Business units) Wageningen UR !
Recommended