Embed Size (px)
Citation preview
“Alt” is German for “old” It’s &me to stop talking about metrics in terms of “alterna&ve” metrics.
Todd A. Carpenter
Executive Director, NISO @TAC_NISO
March 30, 2015
! Non-‐profit industry trade associa&on accredited by the American Na&onal Standards Ins&tute
! Mission of developing and maintaining technical standards related to informa&on, documenta&on, discovery and distribu&on of published materials and media
! Volunteer driven organiza&on: 400+ contributors spread out across the world
! Responsible (directly and indirectly) for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN
About
March 30, 2015 2
1980s Alternative
John Taylor, Duran Duran
1920s Alternative
2010s Not-so Alternative
Cole Plante
Since Jason Priem coined the term “altmetrics”
• There have been some 1,170 ar&cles and posts (per Google Scholar) on the topic of altmetrics
• There have been eight altmetrics conferences • There are now at least four companies focused on providing altmetrics-‐style analy&cs
March 30, 2015 6
When it becomes a standard, its not alterative any more…
But what’s the value in coun&ng Tweets and
Facebook likes?
March 30, 2015 8
More than just popularity
Research is poin&ng to the fact that there is a modest posi&ve correla&on between early-‐signal metrics (altmetrics) and later-‐signal metrics (cita&ons)
Do altmetrics correlate with cita6ons? Extensive comparison of altmetric indicators with cita6ons from a mul6disciplinary perspec6ve by Rodrigo Costas, Zohreh Zahedi, Paul Wouters Do Altmetrics Work? TwiBer and Ten Other Social Web Services by Mike Thelwall , Stefanie Haustein, Vincent Larivière, & Cassidy R. Sugimoto (2006) Earlier web usage sta&s&cs as predictors of later cita&on impact by Brody T, Harnad S & Carr L
March 30, 2015 9
More importantly: Scholarship today is much more than journals
March 30, 2015 10
How do we measure the impact of these different
forms of output?
March 30, 2015 11
Would a researcher focus on only one data source or methodological approach?
March 30, 2015 12 Duke University -‐ Informa&on Ini&a&ve at Duke (IID)
There aren’t metrics and “altmetrics”; there are only metrics!
March 30, 2015 13
We have been using non-‐cita&on-‐based metrics for decades
March 30, 2015 14
March 30, 2015 15
March 30, 2015 16
Traverse Area District Library (TADL)
What has changed is our ability to
collect and analyze data in new ways that are meaningful to others
March 30, 2015 17
What are we lacking?
=
TRUST
=
What are the elements of metrics that we need
to build trust?
Granularity
Time Scale
Identification Definition
Exchange
I omen sound like a broken record
• Defining what is to be counted = standards • How to describe what to count = standards • Iden&fica&on of what to count = standards • Procedures for coun&ng or not = standards • Aggrega&ng counts from network = standards • Exchange of what was counted = standards March 30, 2015 22
Phase 1 Mee6ngs
October 9, 2013 -‐ San Francisco, CA December 11, 2013 -‐ Washington, DC
January 23-‐24 -‐ Philadelphia, PA Round of 1-‐on-‐1 interviews – March/Apr
Phase 1 report published in June 2014 �
Mee&ng Lightning Talks • Expecta&ons of researchers • Exploring disciplinary differences in the use of social media in
scholarly communica&on • Altmetrics as part of the services of a large university library
system • Deriving altmetrics from annota&on ac&vity • Altmetrics for Ins&tu&onal Repositories: Are the metadata
ready? • Snowball Metrics: Global Standards for Ins&tu&onal
Benchmarking • Interna&onal Standard Name Iden&fier • Altmetric.com, Plum Analy&cs, Mendeley reader survey • Twiper Inconsistency March 30, 2015 26
“Lightning" by snowpeak is licensed under CC BY 2.0
March 30, 2015 27
March 30, 2015 28
March 30, 2015 29
30 One-‐on-‐One Interviews
March 30, 2015 30
Lots and lots of ideas
March 30, 2015 31
Alterna6ve Assessment Ini6a6ve
Phase 2 Presenta6ons of Phase 1 report (June 2014)
Priori6za6on Effort (June -‐ Aug, 2014) Project approval (Nov -‐ Dec 2014)
Working group forma6on (Jan -‐ March 2015) Consensus Development (March 2015 -‐ Feb 2016)
Trial Use Period (Feb 15 -‐ May 16)
Publica6on of final recommenda6ons (Aug 16)
White Paper Released
March 30, 2015 33
200 + Ideas 25 Poten6al Work Themes
Defini6ons Applica6on to types of research outputs Discovery implica6ons Research evalua6on Data quality and gaming Grouping, aggrega6ng, and granularity Context Adop6on
March 30, 2015 34
March 30, 2015 35
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of liple importance
Moderately important
Important
Very important
Community Feedback on Project Idea Themes
n=118
March 30, 2015 36
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of liple importance
Moderately important
Important
Very important
n=118
Community Feedback on Project Idea Themes
Top-‐ranked ideas (very important & important >70%)
• Develop specific defini&ons for alterna&ve assessment metrics. (87.9%)
• Promote and facilitate use of persistent iden&fiers in scholarly communica&ons. (82.8%)
• Develop strategies to improve data quality through normaliza&on of source data across providers. (80.8%)
• Iden&fy research output types that are applicable to the use of metrics. (79.8%)
• Define appropriate metrics and calcula&on methodologies for specific output types, such as somware, datasets, or performances. (78.1%)
• Explore crea&on of standardized APIs or download or exchange formats to facilitate data gathering. (72.5%)
• Research issues surrounding the reproducibility of metrics across providers. (70.7%)
March 30, 2015 37
Launching three working groups
March 30, 2015 38
A) Development of defini&ons and descrip&ons of use
B) Defini&ons for appropriate metrics and calcula&on methodologies for non-‐tradi&onal output types. Also work toward promo&on and facilita&on of use of persistent iden&fiers
C) Development of strategies to improve data quality through source data providers
Steering Committee
Project Site: www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group_public/download.php/13295/niso_altmetrics_white_paper_draft_v4.pdf
March 30, 2015 39
Alterna6ve Assessments of our Assessment Ini6a6ve
White paper downloaded 7,400 6mes 21 substan&ve comments received
120 in-‐person and virtual par&cipants at the mee&ngs These 3 mee&ngs apracted >400 RSVPs for live stream
Goal: generate about 40 ideas, in total, generated more than 250 Project materials downloaded more than 22,000 &mes
More than 550 direct tweets using the #NISOALMI hashtag Survey ranking of output by 118 people
Six ar&cles in tradi&onal news publica&ons 15 blog posts about the ini&a&ve
For more
Project Site: www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group_public/download.php/13295/niso_altmetrics_white_paper_draft_v4.pdf
March 30, 2015 41
Questions?
Todd Carpenter
Executive Director [email protected]
@TAC_NISO
National Information Standards Organization (NISO) 3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA +1 (301) 654-2512 www.niso.org
March 30, 2015 42
