Upload
tiffany-rice
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
SCHOLARLY IMPACT
METRICSAN OVERVIEW
JOHAN BOLLEN – [email protected]
INDIANA UNIVERSITY
SCHOOL OF INFORMATICS AND COMPUTING
CENTER FOR COMPLEX NETWORKS AND SYSTEMS RESEARCH
OAI8 - June 2013
OAI8 - June 2013
OAI8 - June 2013
SCIENCE: IDEAS NOT BRICKSScience and scholarly communication matters.
1) Economic and cultural value is enormous, and rests on considerable investments of
1) Capital
2) Infrastructure
3) Human resources
4) Education
2) Outcomes: ideas and information
1) Not the amount of paper pulp produced, number of bricks laid, metal forged, tractors built, fields plowed
2) It’s largely about the ideas and how they are communicated, BUT:
1) Not all ideas matter equally
2) Not all ideas should be communicated
OAI8 - June 2013
SCIENCE AS A GIFT ECONOMY
Gift economy:
- services and good are shared freely without implicit or explicit expectation/agreement of reciprocation
- “economy of abundance, not scarcity”
- found in some societies
Science is a little like that:
- information is shared as freely as possible through publications
- information is perishable (half-life of good idea)
- reward for sharing is essentially a social phenomenon: “esteem”, “prestige”, “influence”
OAI8 - June 2013
IMPACT ~ PUBLICATION
Scholarly outcomes and ideas are traditionally perceived to be mainly shared through the peer reviewed literature, aka publications
- An entire industry has emerged to support this modus operandi
- Not universal, has not always been that way, might not always be this way, but presently dominant
Our ideas of scholarly impact is now strongly tied to scholarly publications
- Ideas that impact or influence fellow scholars reach them via peer-reviewed publications
- Influence and impact is thus expected to be expressed through the medium of peer-reviewed publications
-> Citation data has become de facto currency of impact or influence:
• When one scholar cites the work of another, this is deemed recognition of their influence
• Measuring impact from citations
OAI8 - June 2013
CITATION DATA
OAI8 - June 2013
CITATION NETWORKS
Maps of random walks on complex networks reveal community structure Martin Rosvall*,† and Carl T. Bergstrom*, PNAS 105(4), 1118-1123
The map equation M. Rosvall , D. Axelsson , and C.T. Bergstrom, European Journal of Physics, 178, 13–23 (2009)
OAI8 - June 2013
FROM CITATION DATA TO JOURNAL IMPACT FACTOR
Impact Factor = mean 2 year citation rate
20032001 2002
Journal x All (2003)
OAI8 - June 2013
THAT CONCLUDES THIS LECTURE
Thank you for your undivided attention.
OAI8 - June 2013
HOLD ON
It’s just not that simple
OAI8 - June 2013
A FEW THINGS LEFT TO DISCUSS…
OAI8 - June 2013
“THE MAP IS NOT THE TERRITORY”
Impact, influence is a social phenomenon
• It already exists in the scholarly community
• Most scholars already have a notion of which ideas, publications, journals, and authors matter the most
To measure this social construct of scholarly impact we can choose many different “operationalizations”/measurements:
• Ask scientists: surveys, questionnaires, awards
• Correlates: funding decisions, publication data, citation data
• “Behavioral” data: readership, ILL, reshelving download data, Twitter mentions, etc.
OAI8 - June 2013
MANY PERMUTATIONS
1. Data type and which community it represents
• Citation data: authors
• Usage data: authors, readers, public
• Social media data: everyone
2. Type of metric calculated from (1)
• Counts, normalized counts
• Social network metrics
• Trend metrics
3. Level of granularity:
• Entities: authors, journals, articles, teams, countries
• Time: 5-year span, 2 year span, etc.
OAI8 - June 2013
METRICS, CUBED
Data
type
Metric type
Granu
larit
y
citation
usage
Social mediaau
thor ar
ricle jo
urna
l
coun
ts
Social
net
work
trend
s
BACK TO CITATION DATA AND NETWORKS
Johan Bollen, Herbert Van de Sompel and Marko A. Rodriguez. Towards usage-based impact metrics: first results from the MESUR project, JCDL 2008, Pittsburgh, PA, June 2008. (arXiv:0804.3791v1)
OAI8 - June 2013
- Author-level metrics:
- Total citations- H-index:
- Nth publication with at least n citations (rank order pubs by decr. Cites)
- g-index, e-index, a-index- Co-author network indicators
- Article level metrics:
- Total citations- Normalized citation counts
- Journal level:
- Impact factor- SNIP, Crown indicator- Social network metrics from citation
network (next slide: PageRank, Eigenfactor, Y-factor, betweenness, etc)
CITATION-BASED METRICS
Radicchi et al . (2008) PNAS 105(45) 17268-17272
Hirsch (2005) PNAS 102(46) 16569-16572
Degree• In-degree• Out-degree
Shortest path• Closeness• Betweenness
Random walk• PageRank• Eigenvector
INNOVATION I : CITATION-BASED SOCIAL NETWORK METRICS
OAI8 - June 2013
SOCIAL NETWORK ANALYSIS
PAGERANK FOR JOURNALS
2003 JCR, Science Edition5709 journals, L=0.85
Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics. Information processing and management, 12(5), 297-312.Chen, P., Xie, H., Maslov, S., & Redner, S. (2007). Finding scientific gems with Google. Journal of Informetrics, 1(1), arxiv.org/abs/physics/0604130.
POPULARITY VS. PRESTIGEOutliers reveal differences in aspects of “status”
IF ~ general popularityPR ~ prestige, influence
Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel. Journal status. Scientometrics, 69(3), December 2006 (DOI: 10.1007/s11192-006-0176-z)
Philip Ball. Prestige is factored into journal ratings. Nature 439, 770-771, February 2006 (doi:10.1038/439770a)
PAGERANK FOR JOURNALS
OAI8 - June 2013
Scholarly community and communication is moving online.
Data pertaining to online activities (implicit, behavioral) vs. citation data (explicit declaration of influence)
INNOVATION II: “BEHAVIORAL” DATA
Scholarly community
Scholarlycommunication
items
metrics
Behavioral data
Bibliographic data
Citation
OAI8 - June 2013
BEHAVIORAL DATA
Reading/usage statistics
• Interlibrary loan data
• Reshelving data
• Online catalogue systems
Daily, weekly, monthly access or reading statistics
Usage data:
• Web server logs
• Link resolver data (SFX, etc)
Detailed data on “who”, “what”, “where”, “when”: ability to track scholarly activity in real-time
OAI8 - June 2013
USAGE STATISTICS
COUNTER: member organization defining an auditable standard for reporting and aggregating monthly usage statistics (www.projectcounter.org)
• Journal and article level
• Initiative to define “usage factor”
PLoS Article Level Metrics
• Download numbers
• download trends
OAI8 - June 2013
MESURAndrew W. Mellon and NSF funded project at LANL Digital Library Research and Prototyping and Indiana University
- Very large-scale usage data from publishers, aggregators, and library consortia
- Metrics of scholarly impact derived from aggregated usage data
- Mapping scientific activity from log clickstream data
- Examine“scholarly impact” itself (more later!)
Notable distinction: use of log data that contains clickstream enables metrics and analysis beyond level of usage statistics
Presently concluding planning process (Andrew W. Mellon funded) to evolve to community-supported, sustainable entity
OAI8 - June 2013
INNOVATION III: ALT-METRICSBehavioral AND “attention” data.
• Social media attention, bookmarking, mentions
• Attempt to also capture “social” attention or public impact of scholarly work (not just articles!), another possible dimension of impact
OAI8 - June 2013
SOME RELEVANT RESEARCH
Shuai X, Pepe A, Bollen J (2012) How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations. PLoS ONE 7(11): e47523. doi:10.1371/journal.pone.0047523
Eysenbach G (2011) Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research 13: e123.
OAI8 - June 2013
TWITTER MENTIONS ~ DOWNLOADS, CITATIONS?
OAI8 - June 2013
TWITTER MENTIONS CORRELATE WITH DOWNLOADS AND CITATIONS!
OAI8 - June 2013
ALT-METRICS AS PART OF IMPACT ASSESSMENT
OAI8 - June 2013
CITATION DATA, METRICS, IMPACT, ALT-METRICS, USAGE DATA, LET’S STEP BACK FOR A SECOND
OAI8 - June 2013
BLIND MAP-MAKERS
Odd, nearly tautological situation:
- We have many different metrics or ways to measure impact.
- But no formal or consistent definition of scholarly impact.
- No idea of what exactly impact is, how it manifests itself, what its structure is, along which dimensions it varies. etc
- Whether our metrics actually measure or represent impact
- Our metrics ARE the definition of “impact”
OAI8 - June 2013
SCHOLARLY IMPACT
Metric 4
Metric 3
Metric 2
Metric 5
Metric 6
Metric 1
impact
Not quiteimpact
Some form of impact
Not impact
Validity & Reliability
OAI8 - June 2013
MAPPING OUT IMPACT, ONE METRIC AT A TIME
• Bollen J, Van de Sompel H, Hagberg A, Chute R (2009) A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022
• Priem at al. Altmetrics in the wild.
• Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841
• PLoS ONE alt-metrics correlations: investigated by L Juhl Jensen, Novo Nordisk Foundation
• Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). …A comparison of nine different variants of the h index using data from biomedicine. JASIST, 59(5), 830-837.
OAI8 - June 2013
FINALLY… WHY?Just like social status, scholarly impact (or other) is an interesting scientific study area. It emerges from the scholarly communication process.
BUT pure science is clearly not the only motivation:
• Metrics used in assessment
• Decision-making: funding, promotion, …
• Information filtering
Some of these applications are tremendously useful and potentially enabling of radical changes in scholarly communication, e.g. information filtering and assessing broader community impact of scholarly work.
OAI8 - June 2013
HOWEVER…
Assuming that scholarly impact exists, independently of whether we measure it or not:
Why measure it at all in cases where the scholarly community truly has decision-making power, autonomy? Isn’t the latter a more desirable option than administrators, politicians, and bureaucrats making decisions on the basis of numbers they don’t understand?
So buy me a beer and ask me about our crazy crowd-sourced funding idea…
Johan Bollen, David Crandall, Damion Junk, Ying Ding, Katy Boerner. Collective allocation of science funding: from funding agencies to scientific agency. http://arxiv.org/abs/1304.1067
OAI8 - June 2013
THANK YOU