2015-05-19 Best practices SUHF Stockholm file19.05.2015 · Conditionsfor transformation 1. The citing paper must be indexedin the citation index 2. Normally, alsothe cited paper (thatreceivesthe

1

The citation

Indicator of Quality or performative agent?

2

Science Citation Index

• Introduced in Science 1955 by Eugene Garfield • The citation as a construction

• Citation ≠ reference

Garfield, E. 1955. Citation Indexes for Science: A New Dimension in Documentation through Association ofIdeas. Science 122 (3159):108-111

Time Citation

Reference

3

From cited reference to citation

Cited reference in text:

Garfield (1955) argued that the citation index should be viewed as an ”association ideas index…”

Entry in reference list:

Garfield, E. 1955. Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. Science 122 (3159): 108-111

Citation in the citation index:

GARFIELD E, 1955, SCIENCE, V122, P108

Author Year Source Volume First page 4

Conditions for transformation

1. The citing paper must be indexed in the citation index

2. Normally, also the cited paper (that receives the citation) must be indexed.

3. The cited reference (in the citing paper) must be given correctly so that the reference could be matched to the cited paper.

5

The citation as an indicator of quality?

• Eugene Garfield (1963):• ”One purpose of this communication is to record my forewarning

concerning the possible promiscuous and careless use of quantitativecitation data for sociological evaluations, including personel and fellowship selection”

• ”Impact is not the same as importance or significance”

• At the same time, he also argued SCI to be used to evaluate Journal performance

• Journal Impact Factor (JIF)

6

Garfield, E. 1963. Citation Indexes in Sociological and Historical Research. American

Documentation 14:289-91.

Caution!

The warning reads: ”CAUTION! Any attempt to equate high frequency of citation with worth or excellence will end in disaster; nor can we say thatlow frequency of citation indicates lack of worth.”

Kessler, M.M., and F. E. Heart (1962) ’Concerning the probability that a given paper will be cited’, Report (Massachusetts Institute of Technology, Cambridge).

7

Key arguments for using citations for evaluation

Classic debate:

• Citations as influence vs.

• Citations as indicator of rhetorics/persuasion

Arguments against ”indicator of research quality:

• Citer motivations:• Negative citations

• Perfunctory

• Redundant

• But of course also:• Conceptual/operational

• Evolutionary or

• Confirmational

(from a classification by Moravcsik and Murugusan, 1975)8

Argument for the use of citation analysisas a quality indicator:

”The observation that citations indicate use, and therefore usefulness as well as impact, is the basic argument for using them as an indicator of quality.”

Gläser, J., & G. Laudel. 2008. The Social Construction of Bibliometric Evaluations. In The ChangingGovernance of the Sciences, edited by R. Whitley and J. Gläser. Dordrecht: Springer. 101-123.

9

”being cited”

11

However, most papers are not read at all. No matter what a paper did to the former literature, if no one else does anything else with it, then it is as if it never existed at all. You may have written a paper that settles a fierce controversy once and for all, but if readers ignore it, it cannot be turned into a fact, it simply cannot. (Latour, 1987, p. 40)

Citations as performativity - “being cited”

Traditionally:

• Citations as reward, (passive):• Citation Index as representation of publication patterns

My proposal: Performativity of “being cited”• What research work do citations do?• Citations as construction and epistemological networking

• The citation viewed as an outcome of active achievement or ”performance”

• Reflexive actors (researchers are active)

Citation index as a performative arena

• for publishers, authors, citers, publications and articles; indeed the whole ”citation culture” (Wouters, 1999)

• Authors actively position themselves by choosing journal/field to publish in & research problems to publish on

• Making themselves “cite-able”12

Library cards as usage index

13Left: Sarah Altendorf. "109 | 365 September 15, 2011" Flickr (CC BY-NC 2.0)Right: Jacob Deatherage. "lib_card_all" April 21, 2010 Flickr (CC BY-NC 2.0)

Part 2: The citation as symbol

Means for identifying ”visibility” of research

14

16

Garfield: How to use the Citation Index (1967) - Informersial

C.f. Small, H. (1977): Cited Documents as Concept Symbols. Social

Studies of Science, 8, 327-340

WoS: Citation report profiles

Publication analysis

0

100

200

300

400

500

600

700

800

900

1000

0

50

100

150

200

1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013

Engineering All SCI-e & CPCI-S

1 2 3

Searches using variants of ‘*terroris*’ in title, abstract or author generated keywords.Research area: Engineering

Document types: Research, conference and review articles; editorial material

Indexes=SCI-EXPANDED, CPCI-S Timespan=1989-2013

Selection: 1989-2013: 1842 papers

1. 1989-20002. 2001-20063. 2007-2013

”Internal and external relevance”

19

Relevance quota LCS/GCS

20

# Date / Author / Journal LCS GCS RQ Box

15527 DAVIS FD

PERCEIVED USEFULNESS, PERCEIVED EASE OF USE, AND USER ACCEPTANCE OF

MIS QUARTERLY. 1989 SEP; 13 (3): 319-340

6216 SMALL H

COCITATION IN SCIENTIFIC LITERATURE - NEW MEASURE OF RELATIONSHIP

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 1973; 24 (4):

30551 Venkatesh V, Morris MG, Davis GB, Davis FD

User acceptance of information technology: Toward a unified view

MIS QUARTERLY. 2003 SEP; 27 (3): 425-478

33181 Egghe L

Theory and practise of the g-index

SCIENTOMETRICS. 2006 APR; 69 (1): 131-152

16818 KUHLTHAU CC

INSIDE THE SEARCH PROCESS - INFORMATION SEEKING FROM THE USERS

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 1991 JUN; 42

7361 PRICE DJD

GENERAL THEORY OF BIBLIOMETRIC AND OTHER CUMULATIVE ADVANTAGE

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 1976; 27 (5-6):

13071 DERVIN B, NILAN M

INFORMATION NEEDS AND USES

ANNUAL REVIEW OF INFORMATION SCIENCE AND TECHNOLOGY. 1986; 21: 3-33

23820 White HD, McCain KW

Visualizing a discipline: An author co-citation analysis of information science, 1972-

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 1998 APR; 49

10131 WHITE HD, GRIFFITH BC

AUTHOR COCITATION - A LITERATURE MEASURE OF INTELLECTUAL STRUCTURE

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 1981; 32 (3):

10551 BELKIN NJ, ODDY RN, BROOKS HM

ASK FOR INFORMATION-RETRIEVAL .1. BACKGROUND AND THEORY

JOURNAL OF DOCUMENTATION. 1982; 38 (2): 61-71

53%

9%

37%

37%

46%

47%

42%

34%

36%

9 145 317

10 144 393

7 162 387

8 162 347

5 165 464

6 163 473

3 191 2168

4 191 359

1 322 4238 8%

2 251 683

HistCite1: citation maps (WoS data)

Time

⃝ Nodes: cited documents↗ Arcs: references

1. http://www.histcite.com

Citation kinds

2222

23

Citation kinds

Bibliographic coupling of sources

25

~3500 papers having”subj*well?being*” in title, abstract or keywords 1970-2013

Kullenberg, C. & Nelhans, G. 2015. The happiness turn? Mapping the emergence of “happiness studies” using cited references. Scientometrics 103(2), 615-630.

Bibliographic coupling of sourcesQualitative clustering

Work conducted w/ Christopher Kullenberg, GU: Co-production of Happiness research and society

Part 3: Performance basedresource allocation

Bibliometric indicators at three levels: national, university and individual

27

Why allocate resources based on indicator models?

• Research policy needs tools to allocate fundswithout steering research directly.

• Though, there is also a tradition of directly fundingsector research (not treated here).

• Evaluation based on quality

• Based on the Mertonian CUDOS norms

• Prerequisites:• ”objective measure”• ”unobtrusive measures”• Quantitative models are (quite) easy to operationalize.

28

Possible allocation system models at national level (~2007)

• Australia: count of ISI-publications• Thought to lead to impoverishment of research: Salami slice

publishing; Least Publishable Unit (LPU)

• UK: RAE, panel based evaluations. Since 2014: REF• ”peer review”: expensive and resource-intensive.

• Norway: impact based on publication ”channel” and ”qualitylevel”

• Impact factor, degree of internationalization. Measures at the bookpublisher- or journal level. Expected, not ”real” impact.

• Sweden: Field normalized citations and Waring distributions of publications

• Only WoS-indexed journal articles - low coverage in some areas.

29

’Norwegian model’ (variants also used in Denmark and Finland)

Publikationskanal Nivå 1 (80%) Nivå 2 (20%)

Monografi 5 8

Artikel i periodika eller serier 1 3

Artikel i antologi 0,7 1

Nivåindelning

Publikation

er som kan

nomineras

Kvalitetsmått

Grupp A:

naturvetenskap, medicin

TidskrifterImpact factor +

substitution

Grupp B:

ekonomi, informationsveten-skap, teknik

Alla-”- +

supplement

Grupp C

”mjuk” samhällsvetenskap humaniora

Alla-”- +

supplement

Indikator Vekt

Doktorgradskandidater 30 %

EU-midler 20 %

Forskningsmdsmidler 20 %

Publiseringspoeng 30 %

Two dimensions:

• publication channel

• level of the channel• (0: not scientific)• 1: ordinary scientific• 2: highly prestigious

publication channels

Arguably:• ’Secondary peer review’

• ’Impact factor’ based system

30

Sweden: present performance based fundingmodel (2008/2012)

Basic funding (80 %)

Performance based share (20 %)

1. External funding (50 %)

2. Publication performance (50 %) as normalizeddata for publication & citation rates

Main features

– Four year moving average

– Author fractionalization

– Normalization:– Publications: Waring Distributions

– Citations: Field Normalized Citation Level

– Additional WeightingMedicine + Technology: 1.0; Science: 1.5; Social Sci + Humanities 2.0; Other: 1.1

Basic

funding,

(80 %)

External funding, (50 %)

Publi-cations & citations,

(50 %)

Perfor-

mance

based,

(20 %)

Sources: Prop. 2008/09:50. ’A boost for research and

innovation; Prop. 2012/13:30. ’Research and innovation’

Utbildningsdepartementet [Ministry of Education and Research]. Stockholm: Fritzes.s

31

Comparison Sw/No model

Swedish model

• Transparency:• Variables in the calculated

model are relative

• Selection:• Only published material that is

indexed in WoS ISI

• Measure of quality:

• Citation measures, field normalized

• Source of data:• Already available data (WoS ISI)

Norwegian modell

• Transparency:• Pre-determined ’point system’

• Selection:• More research channels

(Monographs, conf. Proc, journal articles)

• Measure of quality:• ”Secondary peer review”

• Sources of data: • An authorization index must be

created (Cristin, NSB) and publication lists must be updated.. 32

The ’problem of the humanities’

Citing practices differ and are not comparable

• between different disciplines, e.g. natural sciences, social sciences & humanities

There is order of magnitude

• handled by weighting (normalization, fractionalization…)

But could these be compensated for?• By quantitative measures? • or qualitative measures?

33

Motives for weighting

- ”We have made some runs when it comes to what effects different variants of the allocationsystem would give. /…/- I can only say that a big problem for me was that this system – pure and naked – would turn out negative for the humanities and social sciences. We introduced this doubling factor ’to make sure to have a cupped, a protective hand, especially for the humanities.’- We'll see how it goes. Our assessment is that this multiplier 2 is sufficient to protect the humanities. I can not guarantee that it is so. Of course, we will follow up on it. But I think it will turn out positive. - (Applause). ”

Lars Leijonborg (Minister for Education): (translated) excerpt from the transcript of the parliamentary debate before the the voting of the government research bill, Prop 2008/09:50. Internet: http://web.archive.org/web/20100719173732/http://www.riksdagen.se/webbnav/index.aspx?nid=101&bet=2008/09:64 (accessed 2012-05-18) 34

Research and innovation bill 2013-2016’Research and innovation’ (Prop. 2012/13:30)

Key points:

• Performance based share doubled (20 %) from 2014

• ’Peer review’ instead of bibliometrics?

• Cf, the British RAE/REF system or Univeritetskanslerämbetets ”kvalitetsutvärderingssystem för högre utbildning”

• But, implemented ”not before 2018”

• Meaning:• two general elections (2014, 2018)• one research and innovation bill (promised in 2016)

• …will pass before the new model is implemented.

35

Vetenskapsrådets FOKUS-modell

• Vetenskapsrådet: “Forskningskvalitetsutvärdering iSverige”

• December 2014

• “Peer review”, 6-year interval

• Jfr “REF”

• SwePub

70%

15%

15%

Vetenskaplig kvalitet

Kvalitetsutvecklandefaktorer

Genomslag utanförakademin

Vetenskapsrådet (2014b). Modell för fördelning av forskningsresurser till lärosäten (Fokus). http://vr.se/omvetenskapsradet/regeringsuppdrag/regeringsuppdrag/modellforfordelningavforskningsresursertilllarosaten.4.5a947f0d145b21c1709a7.html

36

37

“The Budget 2015 proposes no changes in the distribution of funding for research and postgraduate education”

“I budgetpropositionen för 2015 föreslåsinga förändringar i fördelningen avanslagen till forskning och utbildning påforskarnivå”

(New government budget autumn 2015, p. 192, my translation)

Budget 2015: “The quality spin that disappeared”

Total renegotiated distributoins (TOM) 2010-15

38

LärosäteOmförd.

2010

Omförd.

2011

Omförd.

2012

Omförd.

2013

Omförd.

2014

Omförd.

2015Line Column Win/Loss

Uppsala universitet 685 -5,616 -6,335 -2,221 -1,034 953Lunds universitet 20,043 -4,381 -5,148 -4,500 11,622 2 650

Göteborgs universitet -4,947 1,289 581 3,809 -1,189 −7 483Stockholms universitet -11,258 2,724 1,857 -445 -2,650 −108

Umeå universitet -10,639 -486 2,838 53 -12,611 4,841Linköpings universitet -733 890 1,997 3,137 4,768 394

Karolinska institutet 19,664 1,521 1,013 -1,407 30398 −12,385Kungl. Tekniska högskolan -7,295 -697 -1,355 296 3,920 2,363

Chalmers tekniska högskola 984 -220 -177 996 6,309 i.u.Luleå tekniska universitet -7,517 -58 410 4 -5,117 −1,425

Sveriges lantbruksuniversitet -9,994 -1,734 -234 -3,014 -2,368 5,492Karlstads universitet -5,481 97 431 488 -6,793 2,211

Mittuniversitetet -4,879 1,141 1,609 311 -2,360 -1,626

Linné/Växjö universitet -6,122 1,307 462 -65 -12,229 -127Örebro universitet -460 817 1,132 371 -5,297 -3,317

Blekinge tekniska högskola -2,569 342 221 143 -2,719 -1,213

Högskolan i Jönköping 8,067 161 876 -80 3,244 i.u.Högskolan i Kalmar 2,114 x x x x x

Malmö högskola 3,125 28 490 511 -726 9,068Mälardalens högskola 3,805 411 265 -177 1,004 4,356

Gymnastik- och idrottshögskolan-1,212 332 17 -123 -807 63Borås 2,408 66 143 -35 -1,504 1,834

Dalarna 2,148 206 -130 157 -2,294 -408Gotland 783 61 -235 304 x x

Gävle -649 486 221 -592 -4,335 -1,374Halmstad 2,204 256 -444 1,089 -825 -816

Kristianstad 1,852 666 -691 -302 -2,214 -409Skövde 2,507 -174 -333 763 -791 -369

Väst 1,600 138 26 189 -501 926

Södertörns högskola 1,768 427 493 340 7,098 -3,360Källa: VR, UKV, Utb.dep, Flerpartimotion 2014/15:2839

-250

00

-200

00

-150

00

-100

00

-500

00

5000

1000

0

1500

0

2000

0

2500

0

3000

0

3500

0

4000

0

4500

0

5000

0

5500

0

Uppsala universitet

Lunds universitet

Göteborgs universitet

Stockholms universitet

Umeå universitet

Linköpings universitet

Karolinska institutet

Kungl. Tekniska högskolan

Chalmers tekniska högskola

Luleå tekniska universitet

Sveriges lantbruksuniversitet

Karlstads universitet

Mittuniversitetet

Linné/Växjö universitet

Örebro universitet

Om

förd

elni

ng 2

010

Om

förd

elni

ng 2

011

Om

förd

elni

ng 2

012

Om

förd

elni

ng 2

013

Om

förd

elni

ng 2

014

Om

förd

elni

ng 2

015

Mos

tly

loos

ers

Mos

tly

loos

ers

TO

M:

Un

ive

rsit

ies

20

10

-20

15

3939

Som

e w

inne

rsSo

me

win

ners

TOM: University Colleges 2010-2015

-10000

-5000

0

5000

10000

15000

Blekinge tekniska högskola

Högskolan i Jönköping

Högskolan i Kalm

ar

Malm

ö högskola

Mälardalens högskola

Gym

n- & idr.högsk.

Borås

Dalarna

Gotland

Gävle

Halm

stad

Kristianstad

Skövde

Väst

Södertörns högskola

Omfördelning 2010 Omfördelning 2011 Omfördelning 2012 Omfördelning 2013 Omfördelning 2014 Omfördelning 2015 4040

WinnersWinners

LoosersLoosers NeutralNeutral

University of

Gothenburg ~10 %

Chalmers 4.4 %

Renegotiated shares of Gov’t performancebased funding (2014): ”Bibliometriskt index”.

Universities: University colleges:

6.19 %

University of Borås

0.23 %

41

Aims

1. To map and describedifferent bibliometric models and indicators thatare used in allocation of funds within Swedish HEI:s

• What?• Where?• How?

2. To invite to a criticaldiscussion about the advantages and disadvantages and the relative value of usingsuch indicators for allocation of funds withinacademia.

Gustaf Nelhans & Pieta Eklund; (+ Björn Hammarfelt & Fredrik Åström)

Constant flux of measures at all three levels

• National level (in Sweden):• Field normalized publication and citation measures

• From 2018 suggestion: Peer review – Role of bibliometrics?

• Within (many) universities• Norwegian ”impact factor” model based on secondary peer

review• Swedish citation based model (few Univ’s)• Or combination Swedish/Norwegian models

• Individual level• Norwegian

• H-Index

43

Swedish Academia47 HEIs

Full study:

27 awarding third cycle degrees (doctorates)

In the report:

14 (+3) selected HEI:s

• Size

• Geographic locality

• Wide coverage/specializeduniversities

- Hammarfelt B, Nelhans, G & Eklund, P (2014): The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish Universities. 19th Nordic Workshop on Bibliometrics and Research Policy, Reykjavik, Iceland September 2014- Nelhans, G. & Eklund, P. (2015). Resursfördelningsmodeller på bibliometrisk grund vid ett urval svenska lärosäten. Vetenskap för profession, 30, Borås: Högskolan i Borås

Within universities(empirical data)

Findings - overview

• All universities – with the exception of Chalmersand Stockholm School of Economics- use bibliometric measures to some extent for resource allocation at one or several levels

• The types of measures and models used differs considerably, but models counting publication are more common than citation based models

• The largest and most diversified universities often use a range of measurements depending on faculty


Publication based (10) Citation based (2) Combination of C & P (11)

Blekinge Institute of Technology

Karolinska Institutet* Jönköping University

Halmstad University KTH* Karlstad University*

Linneaus University* Lund University*

Luleå Technical University* Linköping University*

Mid Sweden University* Malmö University

Mälardalen University Swedish University of Agricultural Sciences *

Stockholms University* The Swedish School of Sport and Health Sciences

Södertörn University Umeå University*

University of Borås University of Gothenburg*

University of Gävle Uppsala University*

Örebro University*


* Universities

Point based model: HalmstadUniversity

Publikationstyp Poäng

A Artikel i tidskrift + artikel

forskningsöversikt

8

B Konferensbidrag 1

C Kapitel i bok + del av antologi 2

D Bok (monografi) 8

Tabell 4: Poängsättning för publiceringstyper vid Högskolan i Halmstad

Points in the arts

Publiceringsform Poäng

Bok (eller motsvarande) utgiven på nationellt eller

internationellt förlag

5p

Konstnärligt arbete, refereegranskat 5p

Artikel, refereegranskad (vetenskaplig/konstnärlig) 3p

Konferensbidrag (vetenskapligt/konstnärligt)

refereegranskat

2p

Artikel (vetenskaplig/konstnärlig) 1p

Forskningsöversikt 1p

Kapitel i bok (eller motsvarande) utgiven på nationellt eller

internationellt förlag. Även redaktörskap för bok

1p

Konstnärligt arbete 1p

Tabell 3: Poängsättning för publiceringstyper vid konstnärlig fakultet vid Göteborgs universitet (UR: Regler för registrering avforskning och konstnärliga verk i GU:s open-access-databaser, 2012)

Faculties (9) Departments (16) Individuals (6)

Blekinge Institute of Technology XKarolinska Institutet XJönköping University X (fackhögskolor) XKarlstad University X XKTH X (schools)Linköping university X (Health Science)Linneaus University X X*Lund University XUniversity of Gothenburg XMalmö University X XMid Sweden University XMälardalen University X (research spec)LuleåTechnical University XStockholm University XSwedish University of Agricultural Sciences X X (not formalized)Södertörn University X X (social sciences?)The Swedish School of Sport and Health Sciences

X

University of Borås XUniversity of Gävle XUniversity of Halmstad X (research area)Umeå University X X XUppsala University X* X

- Hammarfelt B, Nelhans, G & Eklund, P (2014): The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish Universities. 19th Nordic Workshop on Bibliometrics and Research Policy, Reykjavik, Iceland September 2014- Nelhans, G & Eklund, P (2014): Resursfördelningsmodeller på bibliometrisk grund vid ett urval svenska lärosäten. Rapport: Högskolan i Borås Maj/oktober 2014

Individuella modeller

Umeå university• Used since 2008 in the

humanities faculty

• individual component

• Researchers and teachers apply for funds in competition

• Publication measures are an important part in the application process

• “Many are of the opinion that the system has negative impact on the work climate”

• “…many cites that they experience individual stress and press.” (Sjögren, 2011)

Linnus university• 2,5 % of the funds are

distributed at the individual level.

• Field normalized publication points

• Publication points that equals between 8.000 – 150.000) SEK:

• Are distributed directly to the researcher.

• Publication points < 8 000 SEK:• Are distributed to the

institution.

• 8.000 SEK = shame limit?

• Excellence share:• 20 per cent of researchers with

highest share of publication points receive an additional 15.000 SEK per individual.

Luleå Techn. Univ.

(award at the department level)

Price tag per published article:

• Level 1: 35.000 SEK

• Nivå 2 or indexed in WoS: 70.000 SEK.

Downside of the performative idiom

’curriculum vitae AND h-index’ ’Gaming the system’Techniques

• self (colleague) citing of references

• editor coercion

• citation cartels

Research policy advice:

Division of Analysis and Evaluation, GU In response to university rankings:

• ”another way of advancing on the list would be to appoint highly cited researchers, since they ’bring with them’ their earlier citations…”

(Division of analysis and evaluation 2013, my translation)

53

False!

54

Från: [____@____.___(utelämnat)]Skickat: den 28 november 2008 15:54Till: [____@____.___]Ämne: Scandinavian Journal of [_____]28-Nov-2008To: Reviewers of manuscripts for Scandinavian Journal of [_____]

Dear Reviewer,We are very grateful for all reviewers´ efforts and contribution to S[_____] and for taking time reviewing manuscripts. Both authors and reviewers are very important for the quality of the journal and without you we will not reach the goal of being the leading international journal of [ämnesområde].We hope you will continue with your excellent work on S[_____] review and contribute to the journal. We would like to further develop the journal and therefore highlight the following issues:

As part of your review, please consider all of the following -1. Manuscripts should not exceed 5000 words excl. abstract, references figures and tables2. Maximum 5 tables and 3 figures are allowed.3. Maximum 12 words in the manuscript title4. Maximum 10 keywords related to the title, and the words should appear in the abstract5. Manuscript should refer to at least one article published in S[_____].6. If you agree to review a paper it is expected that you will also review the revised version of the manuscript. This is important for the quality of the manuscript and for the authors7. Maintain confidentiality throughout the process until publication8. Follow the reviewer’s form and give comments to author and/or the editor9. References should reflect S[_____] international audience10. Be timely. If you are unable to review a paper please let us know immediately as this will fasten the process.Please also take the time to update your keywords, as this will ensure that you are only invited to review papers that fall within your area of expertise.You can do this by logging into your account, clicking on the ‘Edit Account’ tab at the top right hand side of the page, next to the orange ‘Get Help Now’ tab and following the instructions.

Kind regardsEditor-in-Chief, Scandinavian Journal of [_____][_____], professor

Thomson Reuters Highly Cited Scholars

55

Row Labels Total

USA 1699UK 305Germany 165China 150Japan 98Canada 88France 82Netherlands 77Switzerland 67Australia 66Italy 51Spain 43Belgium 32

Saudi Arabia 29Sweden 28Denmark 27South Korea 21Austria 19Finland 14Singapore 14Ireland 12Iceland 11Iran 11Israel 10Turkey 10

Row Labels Total

USA 275Saudi Arabia 152France 53UK 51China 35Australia 30Japan 24Germany 22Spain 21Canada 18Denmark 18Netherlands 18Belgium 15Italy 15

Primary affiliation >10 Secondary affiliation >15“Over three thousand researchers earned the distinction by writing the greatest numbers of reports officially designated by Essential Science Indicators℠ as Highly Cited Papers—ranking among the top 1% most cited for their subject field and year of publication, earning them the mark of exceptional impact.”

3215 Entries

SWE 28 primary affil.

5 secondary affil.

GU: 3

CTH: 1

http://highlycited.com

Consequences for:

59

"Polar Bear A

dF" by Arturo de Frias M

arques -Ow

n work. Licensed under CC B

Y-SA 4.0 via W

ikimedia Com

mons -

http://comm

ons.wikim

edia.org/w

iki/File:Polar_Bear_A

dF.jpg#mediaview

er/File:Polar_B

ear_AdF.jpg

• Research policy:

• the occurrence of bibliometric models have been regarded as a supposedly objective and unobtrusive tool to “tap” the research system for information about its intrinsic qualities, but without influencing the research analysed.

• Unobtrusiveness questioned.

• Research:

• citation based model used in Sweden suit some disciplines better, while others fare worse.

• the introduction of performance-based models creates incentives for researchers to publish according to the yardstick used

• Researchers:• impact down the hierarchy, as performance-based

models have trickled down at all levels in the research practice. In conclusion, the impact on individual researchers is discussed as they grapple with adapting their performance to different and sometimes contradictory quantitative benchmarks.

Performance based models

• Bibliometric indicators: based on historical merits.• Moving averages tend to limit impact of change -> leads to long lag.

• What are the effects on “innovative research”?

• Resource allocation models are (potentially) performative on researchers.

• At the governing level: Audit society, the fear of New Public Management (NPM).

• HEI:s risk loosing self government when decisions are allocated to external funding models: “Hands tied”.

• What is the role of those who are put to do the evaluations?• Bibliometrics function often based in the library.

60

General conclusion

• Bibliometrics in research evaluation:• Quantitative or • Qualitative solutions?• Prevalent both in ’citation’ & ’impact factor’ based models.

• ”Field normalization” and other bibliometric techniques solvesquantitative aspects, but what about qualitative differences in citation practices?

• Individual performativity – incentives to publish• E.g. ”being cited” – how well researchers make themselves cite-able in

citation based metrics.

• ‘Citedness’

61

Literature• Cozzens, S. E. 1989. What do citations count? the rhetoric-first model. Scientometrics 15 (5):437-447.

• Garfield, E. 1955. Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. Science122 (3159):108-111.

• Garfield, E. 1963. Citation Indexes in Sociological and Historical Research. American Documentation 14:289-91.

• Garfield, E. 1972. Citation analysis as a tool in journal evaluation. Science 178:471-79.

• Gläser, J. & G. Laudel. 2008. The Social Construction of Bibliometric Evaluations. In The Changing Governance of the Sciences, edited by R. Whitley & J. Gläser. Dordrecht: Springer. 101-123.

• Gunnarsson, M. 2013. Shanghairankingen 2013. En analys av resultatet för Göteborgs universitet. Enheten för analys och utvärdering PM 2013:07.

• Hammarfelt B, Nelhans, G, Eklund, P (2014): The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish Universities. 19th Nordic Workshop on Bibliometrics and Research Policy, Reykjavik, Iceland September 2014.

• Kessler, M. M. & F. E. Heart (1962) ’Concerning the probability that a given paper will be cited’, Report (Massachusetts Institute of Technology, Cambridge).

• Nelhans, G. & Eklund, P. (2015). Resursfördelningsmodeller på bibliometrisk grund vid ett urval svenska lärosäten. Vetenskap för profession, 30, Borås: Högskolan i Borås.

• Nelhans, G. 2013. Citeringens praktiker. Det vetenskapliga publicerandet som teori, metod och forskningspolitik. Diss: Göteborgs universitet http://hdl.handle.net/2077/33516.

• Small, H. (1977): Cited Documents as Concept Symbols. Social Studies of Science, 8, 327-340.

• Vetenskapsrådet (2014b). Modell för fördelning av forskningsresurser till lärosäten (Fokus). http://vr.se/omvetenskapsradet/regeringsuppdrag/regeringsuppdrag/modellforfordelningavforskningsresursertilllarosaten.4.5a947f0d145b21c1709a7.html.

• Wouters, P. 1999. The Citation Culture. Diss: Faculteit der Scheikunde, Universiteit van Amsterdam.62

Thank you!

[email protected]

63