1

WHAT IS YOUR H-INDEX AND OTHER MEASURES OF ACADEMIC IMPACT Berenika M. Webster, PhD

ULS/ISchool workshop11 November 216

2

Metrics are everywhere

3

They are used to measure/demonstrate impact of…

• National science systems• Research-producing institutions• Research groups• Individual researchers

In order to…• Understand impact of investment• Award funding• Employment and promotion decisions• Apply for funding, job, promotion• Identify experts, collaborators, etc.

4

Learning outcomes• At the end of sessions participants will

• be able to identify paper-level and author-level indicators of impact

• be able to extract these indicators from Scopus, WoS and SciVal

• Understand how individual researchers may use these indicators to demonstrate impact

5

“Citizen” bibliometrics…

• Productivity (publications counts)• leads to “salami slicing”, maybe• quantity vs quality

• Impact (citation counts) • but what impact?

• Impact factor • Speaks to prestige of outlet, not quality of individual paper• 20% of papers in Nature get 80% of all citations (averages do not

work when distributions are skewed) • h-index

• arbitrary, simplistic and lacks consistency• it’s always highest in Google Scholar

6

(BIBLIOMETRIC) STORY OF ONE PUBLICATIONaka article-level metrics

7

Story of one publication

Kato, M., Han, TW., Xie, S., Shi, K., Du, X., Wu, LC., Mirzaei, H., Goldsmith, EJ., Longgood, J., Pei,J., and Grishin, N.V. and Frantz, DE., Schneider, JW., Chen, S., Li, L., Sawaya, MR., Eisenberg, D, Tycko, R. and S. McKnight. (2012) “Cell-free Formation of RNA Granules: Low Complexity Sequence Domains Form Dynamic Fibers within Hydrogels.”, CELL, 149 (4): 753-67. (citations 325, IF 34.242)

8

Story of one publication• Journal characteristics

• JIF or SNIP or another indicator of impact in context• Acceptance rates (e.g. from Cabell's Directory of Publishing Opportunities)

• Co-author characteristics• Names, institutions, countries

• Citations• Number of citations in context (normalised baselines)

• Ratio• Percentile distribution

• Characteristics of citing papers• authors, institutions, countries, subject fields, journals

• Altmetrics • Predictor of citation impact (???)• Indicator of attention• Indicator of impact outside academic realm (e.g. citations in policy documents or clinical guidelines)

9

Story of one publication: WoS

10

Story of one publication: Scopus

11

Story of one publication: altmetrics

12

Story of one publication: more Altmetrics

13

Baselines• Web of Science (WoS) (at Pitt publication and citation data going back to 1980)

• Highly Cited?; Hot?, Impact Factor value

• Essential Science Indicators (ESI) (publication and citation data going back 10 years + current year)• across 22 broad disciplines• Baselines (Averages and Percentiles)

• Journal Citation Reports (JCR)• IF values of thousands of journals and much more (citation networks, rates of obsolescence, etc.)

• Scopus (complete citation data going back to 1996, project to extend to 1970 under way to be completed by end of 2016)

• discipline adjusted citation rates (more granular disciplinary and sub-disciplinary divisions), also adjusted for all doc. types

• SciVal (complete publication and citation data going back to 1996. Based on SCOPUS-indexed publications)

• Over 20 indicators of productivity and impact

14

Story of one publication

My paper has been published in Cell, 2nd top ranked journal in the field of biochemistry and mol. biology and 3nd top title in the filed of cell biology (JCR, 2015 ed.)

To date it was cited 325 times, which places it in the top 1% of all world’s 2012 biochem. and mol. biology papers. The paper has an international and multidisciplinary reach. Citations come from authors from 34 different countries and from across 95 different journals in 35 different fields (including 7 citations from Nature and 6 from Science). (WoS, 20 Sept. 2016).

My paper also attracted attention from non-scholarly audiences, …

Kato, M., Han, TW., Xie, S., Shi, K., Du, X., Wu, LC., Mirzaei, H., Goldsmith, EJ., Longgood, J., Pei,J., and Grishin, N.V. and Frantz, DE., Schneider, JW., Chen, S., Li, L., Sawaya, MR., Eisenberg, D, Tycko, R. and S. McKnight. (2012) “Cell-free Formation of RNA Granules: Low Complexity Sequence Domains Form Dynamic Fibers within Hydrogels.”, CELL, 149 (4): 753-67. (citations 325, IF 34.242)

15

Exercise• Murphy, S.V. and A. Atala (2014) “3D

bioprinting of tissues and organs” NATURE BIOTECHNOLOGY; 32(8):773-785. DOI: 10.1038/nbt.2958

• Describe it using bibliometric indicators discussed today• Citation count• Normalised citation count • Percentile position• Impact of publishing journal• Characteristics of citing publications

• Use WoS or SCOPUS and Altmetric.com

16

Web of Science and Essential Science Indicators

• Murphy, S.V. and A. Atala (2014) “3D bioprinting of tissues and organs” NATURE BIOTECHNOLOGY; 32(8):773-785.

• 337 citations• 337/5.31 = 63.45 (normalised citation impact)• In top 1% of all 2014 Biology and Biochemistry

publications (HICI paper)

17

18

19

(BIBLIOMETRIC) STORY OF ONE AUTHORaka aruthor-level metrics

20

Story of one author

• Source of information about publications• CV• Pitt Faculty Info System (Elements)• ORCID• Google Scholar profile

• How much output is captured in databases• Scopus Author Identifier• ResearcherID (WoS)

21

Scholarly communication practicesSubject area Books and book chapters Conference papers Journal articlesHistory 45.6 3.8 50.6Politics and Policy 43.1 10.8 46.1Language 40.5 7.6 51.8Human Society 31.3 5.6 63Philosophy 29.8 5.4 64.8Economics 27.4 8 64.5Law 26.2 1.9 71.9The Arts 25.2 20.3 54.5Education 21.8 23.6 54.5Architecture 20.8 43.6 35.6Psychology 18.9 4.9 76.2Journalism, library 18.6 24.2 57.2Management 13 34 52.9Earth Sciences 8.6 9.2 82.2Medical & Health Sci 6.6 2.9 90.5Biological Sciences 6.6 2.7 90.7Agriculture 6.3 14.7 79Computing 5 62.3 32.8Mathematical Sciences 5 11.2 83.8Engineering 2.9 45.1 52Physical Sciences 2.7 7.3 90Chemical Sciences 2.3 1.9 95.7

L. Butler, 2006

And what about all other research outputs?

22

Author metrics

• Overview of productivity and impact

• Should use size-dependent variables as these take into consideration volume of outputs• Publications (other outputs) counts and characteristics• Total citation counts in context (discipline, age, output type)• Excellence measures (top 1, 5, 10% of distributions)• ESI’s Highly Cited author?• h-index (if you must) – though it is inconsistent (example later)

23

Story of one author

24

SCOPUS view

Citation per publication rate for my 200 publications indexed in Scopus is 91.2, only 5 of my publications (2.5%) were not cited o date. (Scopus, 8 June 2016)

My research is multidisciplinary spanning across biochemistry, genetics and molecular biology, physics chemistry, materials science and engineering.

I have been publishing consistently since 1982, averaging 7 publications per year in high quality, peer reviewed outlets. These publications, collectively receive upwards of 1,000 citations per year. (Scopus, 8 June 2016)

25

Story of one author

Citations to my publications in the last 10 years placed me in the group of top 1% of biochemistry researchers in the world and 5 of these publications placed in the top 1% of publications in their field. (ESI, 20 Sept 2016)

26

SciVal view - overview

In the last 5 years, 68% of my publications placed in the top 10% of all similar publications (by field and age), and 32% of them were published in the top 10% of scientific journals. (SciVal, 8 June 2016)

27

My publications in biochemistry consistency outperform these of NIH and USA as a whole in relation of impact and percentage of publications in top 1% in the world. (SciVal, 8 June 2016)

28

Whose h-index? One author – 3 different values

137 vs. 143 vs. 160

29

Author 1 Author 2 Author 3

15 150 15

10 100 10

10 50 10

5 25 5

5 5 5

4 1 0

4 0 0

3 0 0

3 0 0

1 0 0

Does not account for:

• insensitive to highly-cited publications

• citation characteristics of publication outlets

• citation characteristics of fields of science

• age of publications

• type of publications

• co-authorship

• self-citations

• scientific age of author

30

More problems with h-index

Ludo Waltman and Nees Jan van Eck, JASIST 2012

If two scientists achieve the same absolute performance improvement, their ranking relative to each other should remain unchanged.

  Scientist X Scientist Y

Publication Citations

1 5 6

2 5 6

3 5 6

4 5 6

5 5 3

6 2 3

7 2 3

  Scientist X Scientist Y

Publication Citations

1 8 8

2 8 8

3 5 6

4 5 6

5 5 6

6 5 6

7 5 3

8 2 3

9 2 3

Scientist X h-index 5Scientist Y 4

Scientist X – 5 Scientist Y - 6

31

Exercise: create a bibliometric profile of a researcher

• Ludo Waltman, Leiden U• Krzysztof Matyjaszewski, CMU• Ervin Sejdic, Pitt• Anne-Wil Harzing, U Melbourne

• Try SCOUPS, WOS and SciVal to find out• Number of publications• Total number of citations• Normalised citations (e.g. within a discipline)• Subject distributions• Characteristics/impact of citing publications• For an entire career span/for last 5 years