OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC ......Blogging • Scopus • SlideShare • SourceForge • Stack Overflow • Twitter • USPTO • Vimeo • Wikipedia • Worldcat

11TH ANNUAL MEETING OF ISMPP 1

1 1 T H A N N U A L M E E T I N G O F I S M P P

OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC APPROACHES

TO MEDICAL PUBLICATIONS April 27–29, 2015

Hyatt Regency Crystal City Arlington, VA, USA



Innovation in Metrics: Capturing the full impact of publications



Key questions in metrics

•The performance of: •My agency (quality of service, cost/value) •My team (quality, efficiency, productivity) •Overall publication plan (volume, reach, understanding, impact) •Individual publications (reach, understanding, impact)

What do you want to measure?

Why do you want to measure it?

What is the best way to measure it?

What are you going to do with that information?




• To show how I performing against comparators • To understand if I should be doing things differently • To defend spend/budget and demonstrate return on investment








• What are the best components of a publication analysis? • The impact factor measures the ‘success’ of the journal, not the

‘impact’ of the article; what article level metrics are important? • How do / can we measure the impact of congress activities as

well journal publications?








• Defend (retrospective) • Apply (prospective)



“NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED”

Jane Grimson. In Blockmans W et al, 2014 Bibliometrics: Use and Abuse in the Review of Research Performance



Publication Metrics:

Practical Application and Experience


INTRODUCTIONS • Moderator

Sarah Feeney, BMedSc Head of Scientific Direction, Complete Medical Communications

• Panel

Bhakti Kshatriya, PharmD Global Scientific Communications, Novartis Pharmaceuticals Corporation Tom Rees, MSc, PhD, CMPPTM

Scientific Strategy Advisor, PAREXEL International Mike Buschman Co-Founder, Plum Analytics


The opinions expressed in this presentation are those of the faculty and do not necessarily represent the opinions of their employers or ISMPP. Neither the faculty nor the facilitator have received honoraria for their participation in this workshop.

DISCLAIMER


OBJECTIVES

•Understand the process of developing publication plan metrics

•Be conversant in the various types of newer publication metrics

•Be educated on which metrics are of most importance to various stakeholders



PUBLICATION METRICS – PHARMA INDUSTRY PERSPECTIVE

Bhakti Kshatriya, PharmD

Global Scientific Communications Novartis Pharmaceuticals Corporation


DISCLAIMER

• This presentation is made at the request of International Society for the Medical Publication Professionals (ISMPP). The presenter, although an employee of Novartis Pharmaceuticals Corporation, is presenting on her own behalf. The information provided and any views expressed during this presentation are those of the presenter and may not be attributed to Novartis Pharmaceuticals Corporation.


WHY ESTABLISH METRICS? • Establish common goals among team members and management

• Helps prioritize publication activities • Facilitate discussion of new publication concepts

─ Does concept align with goals?

• Help ensure appropriate resources for planned publication activities

• Support requests for new budget and/or head count

• Measure progress and identify areas for improvement


CREATING METRICS


FACTORS AFFECTING METRICS Internal Factors • Product life cycle • SOPs & policies • Pub team effectiveness • Pub team resources

– Data generation – Data analyses – Content generation

(authors, writers, contributors, budget)

External Factors • Number of competitors • Author perceptions • Policies at author's

institution • New regulations • Evolving publication

guidelines and journal policies


Publication Metrics

• Volume • Timing • Quality • Topics ~ literature gap analysis


PUBLICATION METRICS: VOLUME

• Number of manuscripts submitted & published • Number of abstracts/posters/oral

presentations at key scientific meetings • Planned vs actual priority submissions/

acceptances • Number of abstracts/manuscripts to regional

meetings/journals


Number of Manuscripts Submitted


• Metric can also be reported by drug/indication • Similar metric can be analyzed for number of manuscripts

published

200

150

100

0

50

Q1 Q2 Q3 Q4

Primary/secondary manuscripts Review articles All Other (editorial, short reports, etc.)



Volume by Type of Publication (journal & congress)

1200

900

600

0

300

2008 2009 2010 2011 2012 YTD

Manuscripts & Review Articles Abstracts Posters/Oral All Other


PUBLICATION METRICS: PLAN VS ACTUAL

40

30

20

15

10

5

0 Actual

Num

ber o

f Pub

licat

ions

Planned Submissions Actual Submission per Plan Actual Submission not Planned for Quarter 35

25

Planned

Q3 Q2 Q4 Q1 Actual Planned Actual Planned Actual Planned


• FDA and EudraCT guidelines – Timing of publications for communication of

clinical trial results • Life cycle management

– May set very specific and/or accelerated timelines for key manuscripts (e.g., new indications, etc.)

• Timelines – When measuring timelines, need to define start

and end date of publication upfront (e.g., end date could be submit, accept, publish)

PUBLICATION METRICS: TIMING


0

Weeks 10 20 30 40 50 60

Drug A Drug B Drug C

MANUSCRIPT SUBMISSION TIMELINESS


PUBLICATION METRICS: QUALITY

• Impact factor (for key publications) – Visibility of journal or article

• Acceptance in first-choice journal • Number of times the article or scientific

statement is cited in other journal articles


PUBLICATION METRICS: JOURNALS

6

5

4

3

2

1

0

1 2 3 4 5

Drug A Drug B Drug C

Journal values La

ncet

Am

J M

ed S

ci

Dia

bet M

ed

Dia

bete

s D

iabe

tes

Car

e D

iabe

tolo

gia

Am J

Med

C

urr M

ed R

es O

pin

Dia

bete

s, O

besi

ty &

Met

abol

ism

D

iabe

tes

Res

Clin

Pra

ct

Dia

betic

Med

icin

e En

docr

inol

ogy

Endo

cr J

In

t J C

lin P

ract

Po

stgr

adua

te M

edic

ine

Ann

Phar

mac

othe

rapy

B

est P

ract

ice

Res

Clin

End

o M

etab

C

lin T

her

Dia

bete

s Te

ch T

her

Endo

cr P

ract

J

Am P

harm

Ass

oc

J C

lin P

harm

acol

J

Man

age

Car

e Ph

arm

Tr

ansp

lant

Inte

rnat

iona

l Tr

ansp

lant

atio

n C

an J

Clin

Pha

rmac

ol

Car

diov

ascu

lar D

iabe

tolo

gy

Dru

gs

Dru

gs T

oday

Ex

pert

Opi

n In

vest

ig D

rugs

Ex

pert

Opi

n Ph

arm

acot

her

Hor

m M

etab

Res

. In

t J C

lin P

harm

acot

her

Leuk

Rec

M

ol C

ell E

ndo

Phar

mac

othe

rapy

Va

sc H

ealth

Ris

k M

anag

Pr

escr

ire In

t


ACCEPTANCE RATE COMPARED WITH JOURNAL AVERAGE

100

80

60

0

30

Overall Journal Acceptance Rate Product-Specific Acceptance Rate (2006-2009)

90

70

50 40

20 10

BJU

Inte

rnat

iona

l

Euro

pean

Uro

logy

Inte

rnat

iona

l Jou

rnal

of

Uro

logy

Inte

rnat

iona

l U

rogy

neco

logy

J

Jour

nal o

f Se

xual

Med

icin

e

Neu

rour

olog

y &

Uro

dyna

mic

s

The

Jour

nal

of U

rolo

gy

Uro

logy

Num

ber o

f Ar

ticle

s


EMERGING PUBLICATION METRICS

• Congress activities – QR codes – number of downloads

• Journal articles – – Online journals

• Number of downloads • Number of clicks through to the article • Number of page views • Number of unique visitors

– Article appearing in the top 25 articles • Specific to journal/publisher

– Number of times cited (Scopus) – Novel and innovative ways of presenting publications


SUMMARY

• Metrics vary and there are more than one that may apply to a given situation

• Metrics are dependent upon a product’s life cycle

• Metrics should have purpose and measure success/effectiveness of the publication plan or team performance

• Emerging metrics may provide new opportunities for measure of publication reach & quality



UNDERSTANDING MODERN IMPACT METRICS

Mike Buschman

Co-Founder, Plum Analytics April 29, 2015


Altmetrics? Article Level Metrics?


Our Approach to Metrics

•Citations •Usage •Captures •Mentions •Social Media

ALLmetrics


Journal citations are the current state of

Scholarly Measure 33

Journal Impact Factor

3.161


Samples of PlumX Metrics Sources

• Amazon • Bit.ly • CrossRef • Delicious • Dryad • dSpace • EBSCO • ePrints • Facebook • figshare

• Github • Goodreads • Google+ • Medwave • Mendeley • PLOS • PubMed

Central • Reddit • Research

Blogging

• Scopus • SlideShare • SourceForge • Stack Overflow • Twitter • USPTO • Vimeo • Wikipedia • Worldcat (OCLC) • YouTube


All Types of Research Artifacts

• Articles • Blog posts • Book chapters • Books • Cases • Clinical Trials • Conference Papers • Data Sets • Figures • Grants

• Interviews • Letters • Media • Patents • Posters • Presentations • Reports • Source Code • Theses /

Dissertations • Videos • Web Pages


Metrics Categories

USAGE (clicks, downloads, views,

library holdings, video plays)

CAPTURES (bookmarks, code forks, favorites,

readers, watchers)

MENTIONS (blog posts, comments, reviews, Wikipedia links)

SOCIAL MEDIA (+1s, likes, shares, tweets)

CITATIONS (PubMed Central, Scopus,

patents)


USAGE

• Is anyone reading our work?

• Did anyone watch our videos?

• Usage is the #1 stat researchers want to know after Citation counts

• PlumX is the only product that includes Usage

clicks, downloads, views, library holdings,

video plays


• Captures indicate that someone wants to come back to the work

• Early indicator of future citations

bookmarks, favorites, readers, watchers

CAPTURES


• This category measures people truly engaging with your research

• Automatically uncover the conversations about your research

• Discover feedback, opinions, etc.

blog posts, comments, reviews,

Wikipedia links MENTIONS


• Social media measures how well a researcher is promoting their work

• This is especially important for early career researchers to measure and understand

• Track the buzz and attention around your research output

+1s, likes, shares, tweets

SOCIAL MEDIA


• Citations still a standard of long-term impact

• Including citations enables side-by-side analysis with other metrics categories

PubMed Central, Scopus, patents

CITATIONS


5 Categories in Action – Example Article

Article level metrics are the building blocks

for PlumX


2015 Article


2014 Article


2013 Article


Researcher Profile


Analytics – document sets and researchers


Journal Comparisons


THANK YOU! Mike Buschman [email protected]

mailto:[email protected]



INTERPRETING ARTICLE METRICS

Tom Rees, MSc, PhD, CMPPTM

Scientific Strategy Advisor, PAREXEL International


Why do we need metrics?

What do we want to achieve

Have we achieved it?

(…and what are the subtle influences on metrics that could trip us up?)


A framework for thinking about metrics

Allen HG, et al. PloS one 8.7 (2013): e68914.


The best articles achieve higher metrics

Bornmann, Lutz (2015): figshare. http://dx.doi.org/10.6084/m9.figshare.1344583 Retrieved 20:54, Apr 18, 2015 (GMT)

http://dx.doi.org/10.6084/m9.figshare.1344583


But the type of article influences metrics differently

Bornmann, Lutz (2015): figshare. http://dx.doi.org/10.6084/m9.figshare.1344583 Retrieved 20:54, Apr 18, 2015 (GMT)



Different topics are picked up differently

Costas R, et al.(2015):. figshare. http://dx.doi.org/10.6084/m9.figshare.1335773 Retrieved 20:04, Apr 19, 2015 (GMT)

Mendeley

Twitter



Document characteristics affect metrics differently

00.10.20.30.40.50.60.70.80.9

1

Pages References Title length Countries

Cor

rela

tion

(r-sq

uare

d)

Citations Blog posts Tweets News

Haustein S, et al. PLoS ONE 2015;0(3): e0120495


Open access increases scholarly activity more than social media

Alhoori, Hamed, et al. "On the Relationship between Open Access and Altmetrics." iConference 2015 Proceedings (2015).


Even within open access, choice of journal influences metrics differently

Rees TJ, et al. Poster presentation ISMPP 8AM 2012


Video abstracts don’t affect views but does that matter?

Rees TJ, et al. Poster presentation ISMPP 11AM 2015


Few researchers currently use social media to discover recommended research papers

0% 20% 40% 60%

Twitter

Facebook

Frontiers

Academia.edu

ResearchGate

LinkedIn

BioMedExperts

Google+

Mendeley

0% 20% 40% 60%

% of full sample* % of service users

* Extrapolation Rees TJ, et al. Oral presentation ISMPP European Meeting 2014


And social media is not just a young-person thing

0102030405060708090

18-24 25-34 35-44 45-54 55-64 65+

%

Age (years)

(NPG), Nature Publishing Group (2014): NPG 2014 Social Networks survey. figshare. http://dx.doi.org/10.6084/m9.figshare.1132584Retrieved 11:36, Apr 24, 2015 (GMT)



All that tweets is not gold…


Dam that Social Networking…


People are consuming this information!

Feeny S, et al. ISMPP 10AM 2014


Conclusion

• Evaluating publication impact is complex, and so requires multiple measures

• Understanding what different metrics indicate is critical to their interpretation

• We are still limited to measuring what we can, rather than what we want to

• Remember, Activity ≠ Impact!



Innovation in Metrics: Capturing the full impact of publications



OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC APPROACHES

TO MEDICAL PUBLICATIONS April 27–29, 2015

Hyatt Regency Crystal City Arlington, VA, USA

Documents

OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC ......Blogging • Scopus • SlideShare • SourceForge • Stack Overflow • Twitter • USPTO • Vimeo • Wikipedia • Worldcat