WISER: Bibliometrics II The Black Art of Citation Rankings Angela Carritt Juliet Ralph March 2011...

WISER: Bibliometrics IIThe Black Art of Citation Rankings

Angela Carritt

Juliet RalphMarch 2011

These slides are available on http://www.bodleian.ox.ac.uk/services/training

/wiser/presentations

Overview of Session• What are bibliometrics?• Why bother?• Problems• Calculating impact factors using Web of Science

• …for an article• …for a researcher• …for a department / institution

• Calculating impact factors using Scopus• Calculating journal impact factors using the Journal Citation

Reports

What are bibliometrics?•…the statistical analysis of books, articles, or other publications. Oxford English Dictionary

•…”ways of measuring patterns of authorship, publication and the use of literature” HEFCE, Bibliometrics and the Research Excellence Framework (REF)

• Use of citation information to measure the impact of research

2008

2010

2009

2010

2010

2008

Citations to an individual paper

Citations to a researcher

Citations to a research

group/department

Citations to a journal

Continued….• Wide range of formulae developed which aim to use

citation information to assess research impact accurately• Some of the things that are sometimes considered…

• Number of articles published• Number of self citations• Type of article• Period over which articles were published• Prestige of citing journal• Subject/discipline…

The power behind

Why bother?... REF• “The pilot exercise showed that citation information is not sufficiently

robust to be used formulaically or as a primary indicator of quality; but there is considerable scope for it to inform and enhance the process of expert review” HEFCE www.hefce.ac.uk/research/ref/Biblio/

• Research Excellence Framework 2014• quality of research outputs (65% weighting 2014)• wider impact of research – (20% weighting in 2014) - measured by

case studies• vitality of the research environment 15% weighting in 2014)

• Expert panels decide whether or not to use citation information under the “quality of research outputs” heading

• Panels to annouce their “criteria statements” in late 2011 - watch this space

HEFCE, REF and citation information• Central guidance to ensure that departments are not

disadvantaged by missing data

• Information will be standardized, simplified…and made available to universities for a small fee

• HEFCE will assess the (financial) cost of using citation information

• More @ http://www.hefce.ac.uk/research/ref/Biblio/

Why bother• Benchmarking of departments and research groups• Grant applications• Recruitment of individuals

Lots of problems...• Self-citations • Negative citations• Insignificant citations• Multiple authors/research groups• Incomplete citation lists - does not include citations in

books... or other publications not indexed by Web of Science/Scopus…poor coverage of conferences

• Not comparable across disciplines – may disadvantage researchers in interdisciplinary fields

• Review articles are more highly cited than original research• More...

http://www.slideshare.net/guest633b30/bibliometrics-and-scientometrics-1065282

© http://ceochef.files.wordpress.com/2008/07/pinch-of-salt.jpg

Individual articles• On Web of Science - use Cited Reference Search• Better than General Search at retrieving Variants (incorrect

citations).

• Example:• Effectiveness of PowerPoint presentations in lectures.

Author(s): Bartsch RA, Cobern KM • Source: COMPUTERS & EDUCATION    • Volume: 41    Issue: 1    Pages: 77-86    • Published: AUG 2003.

Cited reference variant

Analyzing a highly cited article

Analyzing the Citing articles

…by highest citing Authors

…by highest citing Journals

or create a Citation Map

Citation map

Impact factors for a researcher

Citation report for an author

h-index=75

75 articles cited 75 times or more

Citation tracking & analysis in SCOPUS

• Scopus covers 18,000 journals in Science, Mecicine, Social Sciences & Humanities

• Results include journal articles & conference proceedings• Each record for a paper shows the number of times it has

been cited in Scopus since 1996• Similar analytical tools to Web of Science• www.scopus.com

Citation analysis in Scopus

h-index = 73 based on citations in

Scopus post-1996 to 537 papers

h-index = 63 if exclude

self-citations

…for a department or institution• Not always easy!

• Web of Science - Search the Address field• Must use standard abbreviations, e.g:• oxford univ• But word order makes a difference!• univ oxford retrieves thousands more results• more researchers use University of Oxford - not

Oxford University - in their address• Can also search by postcode, e.g. • OX1 2JD (Wellington Square)

Searching by Address

Check the abbreviations lists

Searching by department or college• Postcode or name• Tip: include univ oxford same or oxford same in

search string to weed out other institutions, e.g.• univ oxford same pathol• oxford same trinity• oxford same Ludwig Inst Canc Res

Scopus has Affiliation Search

• Search by institution name not postcode• university of oxford retrieves same results as

oxford university

Scopus Affiliation Search

Scopus Affiliation ID

Symplectic Elements

Symplectic Elements• record keeping tool for research outputs • automatic searching of databases such as Web of Science

& Scopus• Facility to run reports for a department • For more information go to• www.admin.ox.ac.uk/pras/research/symplectic/ • The solution for your department?• Contact symplectic@admin.ox.ac.uk

Impact Factors - Journal Citation Reports (JCR)

• Compare relative importance of journals using citation information

• Based on citation data from Web of Science• Covers

• > 5,900 journals in science and technology• > 1,700 journals in the social sciences

Uses & Abuses• Uses

• Help you to decide where to publish• Help librarians to make decisions about journal purchase etc

• Abuses• Have been used to measure research impact of individual and

research groups

Impact Factor - Number of times the “average” article published in the previous 2 (or 5) years was cited this year.

Calculated:no. of citations to articles published in the last 2 (or 5) years ÷ no. of articles published in same period.

Immediacy IndexMeasures how quickly articles are cited. Calculated: no. of citations to articles published this year ÷ no. of articles published this year.

Cited Half-Life - How many years you have to go back to account for 50% of citations to the journal. e.g. 50% of citations were to articles published in the last 3.5 years. The rest cited earlier articles.

Detailed view

Detailed view continued

Citations TO the journal by year of cited article (e.g. 333 of this year’s citations to Biological Review were to articles published in in 2005 )

Detailed view continued

Citations from Biological Review (to other journals and self cites) by year of cited article E.g. 334 citations from Biological Reviews journal cited articles published in 2007

Detailed view continued

Type of articles included

Eigenfactor Metrics• Take into account prestige

of citing sources • Use “Google style”

algorithms • Attempts to measure how

often the average researcher would encounter the journal

• http://well-formed.eigenfactor.org/

Google’s PageRank from http://en.wikipedia.org/wiki/PageRank

Eigenfactor: Score & Article Influence• Eigenfactor – increases with the size of the journal

• Article Influence – Takes into account number of articles published. More comparable to the JCR impact factor

Journal Impact Factors: ProblemsUse with caution…Results are skewed by many factors… • Size • Frequency / time of publication• Type of content - review articles are more heavily cited than original

research…• Journals that are not indexed by WOS are disadvantaged

• Non English Language journals disadvantaged• “Cited” only journals

• Problems when journals change names • Results are not comparable across discipline (some journals in the

wrong discipline)

• Journal impact factors should NEVER be used to assess impact of researchers / groups etc

Brief bibliographyGeneral• Broadus, R. N., “Towards a definition of Bibliometrics” Scientometrics,

vol. 12, nos 5-6, (1987) 373-379 @ www.springerlink.com/content/v111750n14086384/fulltext.pdf

• HEFCE papers, reports, papers and pilots on the use of bibliometrics in the REF @ www.hefce.ac.uk/research/ref/Biblio/

H-Index• Hirsch, J. E. (15 November 2005). "An index to quantify an individual's

scientific research output". PNAS 102 (46): 16569–16572 @ www.pnas.org/content/102/46/16569.abstract

• Ball, P. “Index aims for fair ranking of scientists”, Nature 2005 Aug 18 436: 900

Brief bibliographyJournal Impact Factors and the JCR• The Thomson Reuters Impact Factor (originally published in the Current

Contents print editions June 20, 1994) @ http://thomsonreuters.com/products_services/science/free/essays/impact_factor/

• Garfield, E. "The agony and the ecstasy: the history and meaning of the Journal Impact Factor“ Paper at the International Congress on Peer Review And Biomedical Publication, Chicago, September 16, 2005 @ http://garfield.library.upenn.edu/papers/jifchicago2005.pdf

Want more…• Wikipedia entries on the following topics include useful bibliographies: the

h-index, journal impact factors and the Eigenfactor

• A Google Scholar (http://scholar.google.co.uk/) search will return many useful articles including subject studies on the use of bibliometrics