BIBLIOMETRICS

BLASPHEMY“the biases and deficiencies of individual citers are repaired to a tolerable degree by the combined activity of the many” (White, 2001), where deficiencies are reduced to “random noise” (Cawkell, 1976) and “…references can be used on the aggregate as an indicator of influence” (Small, 1987)

Introduktion af h-indeks i 2005, førte til en eksplosion af FNI, der hævder at være mere robust, gyldig og sofistikeret end de øvrige.

FNI er ødelæggende, tilskynder til en ‘gaming’ mentalitet og støtter universiteterne i at presse medarbejdere til målrettet at øge deres indeksog tilskynder forskningspolitik der overvåger forskningsresultater på det individuelle niveau (Dahler-Larsen, 2012; Collini, 2012).

Improvements on hhw (weights citations); hn (field comparisons); ht (id priority articles); hpd (seniority, 10 years); hc (seniority, 4 years); Q2 (number/impact papers in h); hα (granular h ranks) hT (alternative h calculation)

Complimentary to hm-quotient (h normalized for age); m (median C in h core); A (average C in h core); R (square root of A); E (effect papers not in h); AR (Citation intensity & age of C); b (self citations, top papers); Rational h (distance to higher h); Hm (multi-authorship); n (field comparison); h sequences & matrices (field comparison); hf (field comparison); Alternative h (multi authorship); Pure h (multi authorship); Adapted pure-h (multi-authorship); Dynamic h (compare peers); hmx (database comparison)

NarrativeIQP (average quality); Index Age & Productivity (academic age on productivity & impact); %HCP (top 20%); Classification of durability (document type & field comparison); DCI (citation age)

Replace hg, gα (more granular than h); mg-quotient, AWCR (normalized for age); rational g (distance to higher g); hg (compare h & g values); h2 (cummulative acheivement); ħ, x (compare field/seniority); ct, at(aging rate); wu(excellent papers); π (production & impact); POP h, AWCRpa (multi-authorship); AW (adjust highly cited papers)

With no advisory boards, common standards or contextual assessments, indicators are mostly incomparable, which in fact impedes the development of the field and makes the users of scientometric results mistrustful. (Vinkler, 1996).

Standarder vedrørende de etiske aspekter af evaluerende bibliometri er igen nyligt blevet foreslået.(Bornmann, 2008; Bach, 2011; Furner, 2014; Hicks et al, 2015)

Contextual bibliometrics1994/1996

Matematiske standarder for

analyse, præsentation og

fortolkning af data2008

Etiske standarder til evaluering af

individer2011

Begrebsramme om etik & bibliometri ift. fordeling af midler

2014

DORA: brug ikke JIF i evaluering af individer

2012

Leiden Manifesto, Metric Tide informeret

brug og formidling2015

Contextual bibliometrics

2016

Evalueringer baseret på indikatorer, kan føre til antagelser om en forskers produktivitet og impact, som kan være udokumenterede, og kan påvirke forskerens selvopfattelse.

10

FNI GIVER ET SNAP-SHOT AF INDIVIDETS IMAGE OG CENTRALE PERSONLIGHEDSTRÆK

SAMMENLIGNINGER KAN EKSPONERE DEN ENKELTE

ALLE FORMER FOR DATA BRUGES TIL AT ØGE FNI, INDIVIDETS VALIDITET INDEN FOR DOMÆNET OG DERES SELVVÆRD FNI BERIGER EN EVALULERING MED

OBJEKTIVITET, REDUCERER KØN, KULTURELLE OG RACEFORDOMME

DOKUMENTERE AT MAN IKKE KLARER SIG BEDRE END ENS KOLLEGAER KAN SKADE FORSKERENS SELVOPFATTELSE

FNI BIDRAGER IKKE ALTID MED VALUE-ADDED INFORMATION; INFORMATION KAN VÆRE REDUNDANT

SUCCES DEFINERES INDEN FOR EVALUERINGSSYSTEMET

EVALUERING

OUTPUT

DATA

MODEL

FORTOLKNING

AFSPEJLERREALITETER

SUCCES I SYSTEM vs SUCCES I FELTET

FEJL/MANGLER

INDEKSERING

UDFORDRINGER

03/05/2023 12

UDFORDRING #1: MODELLERNE

Forstå mekanismer i konstruktionen af bibliometriske indikatorer.Forstå hvad det er, vores data kan vise.

e=18

h=8

m=1

=3CPP=18t=4

S c (i) = γ ∗ (Y (now) − Y (i) + 1) − δ ∗ |C(i)|

Vægtet citations score for en artikel

Evaluerings år minus publikations år

Antal citationer

Valgbar koefficient (sat til 4), som gør at en atikel fra evalueringsåret tildeles en faktor 4

03/05/2023 15

UDFORDRING #2: DATA

”In time-keeping, in trading, in fighting, men counted numbers; and finally, as the habit grew, only numbers counted.”

Mumford, (2010)

03/05/2023 16

Rå publikations/citationstal, fortæller os ingenting om forskningskvaliteten.

Observationer – studerer processer og interaktioner.

Spørgeskemaer – identificerer meninger, erkendelser og forståelse.

Interviews – inviterer til feedback som man normalt ikke vil få.

03/05/2023 17

UDFORDRING #3: INDEKSERING

Indikatorer er i sig selv ikke neutrale! Den største andel af indikatorer og databaser har deres oprindelse i de nordlige lande, og afspejler derfor karakteristika for videnskab i disse regioner.

03/05/2023 18

Hvordan kan vi udvikle lokale systemer for at sammenligne, skabe & gen-sammenligne gen-skabe indikatorværdier i multifacetterede data, så vi undgår ”én-dimensionelle profiler”?

03/05/2023 19

UDFORDRING #4: OUTPUT

Det vi ikke tæller, er usynligt.

03/05/2023 20

½

Hvad er en publikation?Hvad er en citation?Hvad er en forsker?

03/05/2023 21

UDFORDRING #5: EVALUERING

“The assessment itself is completely artificial. It’s not ranking researchers in accordance with their ability to develop, reach their potential, and explore their creative interests. Those things you’re not testing..... it’s a rank that’s mostly meaningless. And the very ranking itself is harmful. It’s turning us into individuals who devote our lives to achieving a rank. Not into doing things that are valuable and important.”

Noam Chomsky (2015)

03-05-2023 22

Mød dem der bliver evalueret.

Undersøg hvorfor de publicerer der hvor de gør.

Argumenter for relevans af indikatorer.

Rådgiv.

Kommuniker/formidl.

CHOOSE WISELYTransparens

Demografi

Motivation

Mangfoldighed

Åbenhed

Multi-dimensional research assessment matrix (Moed, 2011)

UNIT OF ASSESSMENT

PURPOSE OUTPUT BIBLIOMETRIC INDICATOR

OTHER INDICATORS

INDIVIDUAL ALLOCATE RESOURCES

RESEARCH PRODUCTIVI

TYPUBLICATION

S PEER REVIEW

RESEARCH GROUPS

IMPROVE PEFORMANC

E

QUALITY, SCHOLARLY

IMPACT

JOURNAL CITATION IMPACT

PATENTS, LICENCES, SPIN OFFS

DEPARTMENTINCREASE

MULTI-DISCIPL.

RESEARCH

INNOVATION & SOCIETAL

BENEFIT

ACTUAL CITATION IMPACT

INVITATIONS FOR

CONFERENCES

INSTITUTIONINCREASE REGIONAL

ENGAGEMENT

SUSTAINABIL-ITY & SCALE

INT. CO-AUTHORSHIP

EXTERNAL RESEARCH

INCOME

RESEARCH FIELD

PROMOTION, HIRING

RESEARCH INFRASTRUC

TCITATION

”PRESTIGE”PHD

COMPLETION RATES

Hvordan kan tilgængelighed af kontekstuel information forbedres?

Hvordan kan vi vejlede individer i brugen af deres information?

Hvordan opfylder vi løftet om informeret peer review?

Hvordan kan vi influere institutioners tilgang til forskningsevaluering?

03-05-2023 26

Contextualized bibliometrics

Levér information der kan eksploreres.

Undgå for meget vægt på det der let kan kvalificeres/tælles.

Vælg simple indikatorer frem for komplicerede indikatorer.

Vær kritisk. Vær proaktiv.

CALL FOR ACTION

Som bibliometrikere, skal vi forpligte os til at underbygge meningsfulde sandheder.

Lorna Wildgaard, Ph.D Lorna.Wildgaard@hum.ku.dk

03/05/2023 28

REFERENCER

• Bach, J. F. (2011). On the proper use of bibliometrics to evaluate individual researchers. Académie des sciences. Retrieved 23-6-2015 from: http://www.academie-sciences.fr/activite/rapport/avis170111gb.pdf

• Bornmann, L., Mutz, R., Neuhaus, C., and Daniel, H-D. (2008b). Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8(1), 93-102.

• Cawkell, A. E. (1976). Understanding science by analysing its literature. The Information Scientist, 10(1), 3-10.

03/05/2023 29

REFERENCER

• Chomsky, N. (2015) Creative by Nature, Blog post: https://creativesystemsthinking.wordpress.com/2015/02/21/noam-chomsky-on-the-dangers-of-standardized-testing/

• Collini, S. (2012). Bibliometry. In What are universities for? (pp.120-131) London: Penguin.

• Dahler-Larsen, P. (2012). The Evaluation Society. California: Stanford University Press.

• Furner, J. (2014). The Ethics of Evaluative Bibliometrics. In B.Cronin & C. Sugimoto (Eds.), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact (pp. 85-107). Massachusetts: MIT Press.

• Hicks, D. Wouters, P. Waltman, Ludo. de Rijcke, S., and Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429-431.

03/05/2023 30

REFERENCER

• Moed, H. (2011). The multi-dimensional research assessment matrix. Research Trends, 23 (May 2011):http://www.researchtrends.com/issue23-may-2011/the-multi-dimensional-research-assessment-matrix/

• Mumford, L. (ed.2010). Technics and Civilization. University of Chicago Press. p.22

• White, H. D. (2001). Authors as citers over time. Journal of the American Society for Information Science and Technology, 52(2), 87-108.

• Small, H. G. (1987). The significance of bibliographic references. Scientometrics, 12(5-6), 339-341.

• Vinkler, P. (1996). Some practical aspects of the standardization of scientometric indicators. Scientometrics, 35(2), 235-245.