Classifying and visualizing the disciplinary focus of universities: The invisible factor of...

Classifying and visualizing the disciplinary focus of universities

THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS

PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014

Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar

University of Granada

Rationale Background Research questions Contributions Conclusions

Personal motivation

The I-UGR Rankings aim to complement international rankings and account for disciplinary differences.

My contribution to this project is reflected in this thesis.

2

1th edition published in 2010

Rationale Background Research questions Contributions Conclusions

Personal motivation

3

Rationale Background Research questions Contributions Conclusions

Rankings as bibliometric tools

4

[…] rankings tend to emphasize vertical differences between institutions […]. At the same time, they obscure horizontal differences.

Marginson, 2007

[…] ‘mediocrity’ of a university does not necessarily imply that it is mediocre in all disciplines.

Moed, 2008

Rationale Background Research questions Contributions Conclusions

Rankings in research policy

5

International rankings are focused on

World-Class universities

However, many research

managers use them to

evaluate their national system

Rationale Background Research questions Contributions Conclusions

Table of contents

6

Background • Bibliometrics for evaluating

• National Performance research funding systems

• International Rankings

Research questions • Are national university rankings necessary in an international

context?

• How can we develop tools to identify disciplinary profiles?

Contributions • Complementing international rankings

• Mapping techniques for interpreting rankings

Conclusions • Discussion

• Implications

7

Rationale Background Research questions Contributions Conclusions

Bibliometrics as an evaluation tool

The need for bibliometric indicators arises from the enormous growth in scientific and technological activity that has been occuring for centuries, and has now reached avalanche proportions.

Narin et al., 1994

Type of indicator Type of

measurement Unit of analysis

Papers

Citations

Usage

Data

Altmetrics

…

Activity measurement

Impact measurement

Linkage measurement

Micro level Meso level Macro level

8

Rationale Background Research questions Contributions Conclusions

Why the need to assess universities’

performance

I. Research as the Second Mission of universities

II. The expansion of Big Science

III. Knowledge-based economies and the production

of human capital in a globalized context

IV.Economic constraints and the need for

accountability

9

Rationale Background Research questions Contributions Conclusions

National funding systems and

International Rankings

10

Rationale Background Research questions Contributions Conclusions

The problem with international rankings

11

Rationale Background Research questions Contributions Conclusions

The problem with international rankings

Rankings consider universities as homogeneous,

neglecting their disciplinary differences

Health &

Life Sciences

Natural & Exact

Sciences

Social Sciences Humanities

12

Rationale Background Research questions Contributions Conclusions

The problem with international rankings

Can we use international rankings to analyze national systems?

13

Rationale Background Research questions Contributions Conclusions

The problem with international rankings

Thus, the country in which a university is located plays an important role in the performance of a university

Bornmann, Mutz & Daniel, 2013

Countries typically have legal and institutional rules within which their universities function

Aghion et al., 2010

Rationale Background Research questions Contributions Conclusions

Table of contents

14

Background • Bibliometrics for evaluating

• National Performance research funding systems

• International Rankings

Research questions • Are national university rankings necessary in an international

context?

• How can we develop tools to identify disciplinary profiles?

Contributions • Complementing international rankings

• Mapping techniques for interpreting rankings

Conclusions • Discussion

• Implications

Rationale Background Research questions Contributions Conclusions

15

RQ 1. Are national university rankings necessary in an international context?

• An insight into the importante of

national rankings in an international

context

• What do university rankings by fields

rank?

Rationale Background Research questions Contributions Conclusions

16

RQ 2. How can we develop tools for research policy-makers that can allow them to identify disciplinary profiles?

• What do university rankings by fields rank?

• On the use of biplot analysis

• Mapping academic institutions according to their journal publication profile

• Análisis de redes de las universidades españolas

Rationale Background Research questions Contributions Conclusions

Table of contents

17

Background • Bibliometrics for evaluating

• National Performance research funding systems

• International Rankings

Research questions • Are national university rankings necessary in an international

context?

• How can we develop tools to identify disciplinary profiles?

Contributions • Complementing international rankings

• Mapping techniques for interpreting rankings

Conclusions • Discussion

• Implications

18

Rationale Background Research questions Contributions Conclusions

An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities

Research questions > Are national university rankings necessary, considering the

existence of international rankings?

> Levels of agreement between national and international

rankings regarding their coherence and concordance

Methodology > Rankings analyzed: Shanghai Ranking – QS Ranking –

NTU Ranking – Leiden Ranking – I-UGR Rankings

> Indicators: Spearman rank correlation – Coincidence of

top Spanish universities

19

Rationale Background Research questions Contributions Conclusions

An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities

Conclusions

> Spain is poorly represented in international rankings

25% of the Spanish system

> Despite methodological differences, national rankings are

a necessary complement of international university

rankings

> We must consider these methodological differences

derived from the construction of fields as well as the indicator

used

20

Rationale Background Research questions Contributions Conclusions

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

Research questions > Do rankings by fields represent the structure of

universities?

> Can we provide indicators that show the degree of

correspondence between each field and organizational unit?

Methodology > Universities analyzed: Pompeu Fabra – Granada

> Data processing: Identification of organizational units

> Indicators: Betweenness Centrality – Gini Index –

Number of subject – Number of disciplines

21

Rationale Background Research questions Contributions Conclusions

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

Results

Faculties Departments

University of Granada

22

Rationale Background Research questions Contributions Conclusions

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

Results

Faculties Departments

Pompeu Fabra University

23

Rationale Background Research questions Contributions Conclusions

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

Conclusions

> Working with addresses adopting a top down approach has

implicit problems of difficult solution

> The structure and size of universities varies significantly

from one to other which prevent from using organizational

structures to construct fields

> The Gini coefficient and the Betweenness Centrality are

useful indicators for pointing out discrepancies between fields

and organizational units

24

Rationale Background Research questions Contributions Conclusions

How can we identify disciplinary

profiles of universities?

Using mapping techniques to analyze

the multidimensional nature of

universities

25

Rationale Background Research questions Contributions Conclusions

On the use of biplot analysis for multivariate bibliometric and scientific indicators

Objective

> Present the visualization methodology known as Biplot

Analysis

Methodology > Three case studies: European countries – Universities –

Scientific fields

> Validation: Comparison with other visualization

techniques: CA – MDS – PCA

26

Rationale Background Research questions Contributions Conclusions

On the use of biplot analysis for multivariate bibliometric and scientific indicators

27

Rationale Background Research questions Contributions Conclusions

On the use of biplot analysis for multivariate bibliometric and scientific indicators

Comparison with other techniques

28

Rationale Background Research questions Contributions Conclusions

On the use of biplot analysis for multivariate bibliometric and scientific indicators

Case study 2. Universities

29

Rationale Background Research questions Contributions Conclusions

On the use of biplot analysis for multivariate bibliometric and scientific indicators

Conclusions

> Biplot analysis may well be an important analysis tool for

bibliometric studies

> They are easy to read tools of potential use in the research

policy arena

> As well as representing the relations between cases, they

also represent relations between variables, and variables and

cases

30

Rationale Background Research questions Contributions Conclusions

Mapping academic institutions according to their journal publication profile: Spanish universities as a case study

Objective

> Present a novel methodology for representing universities

according to their journal publication profile

Structure > Methodological proposal

> Case study with Spanish universities

31

Rationale Background Research questions Contributions Conclusions

Mapping academic institutions according to their journal publication profile: Spanish universities as a case study

Hypothesis

Universities with a similar research profile should publish in the same scientific journals

32

Rationale Background Research questions Contributions Conclusions

Mapping academic institutions according to their journal publication profile: Spanish universities as a case study

Information and Communication Technologies

Collaboration and geographical proximity

Basic vs. Applied

40-50% Q1 journals

30-40% Q1 journals

<30% Q1 journals

33

Rationale Background Research questions Contributions Conclusions

Mapping academic institutions according to their journal publication profile: Spanish universities as a case study

Conclusions

> Mapping techniques end with limitations derived from a

rigid classification system of universities

> Science maps combined with impact or visibility indicators

shows vertical stratification and horizontal diversification

> The technique highlights geographical, social and/or

historical relationships

> Limitation: Too many links between universities blur

similarities between low perfomance universities

34

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Objectives > Map and analyze the research perfomance of Spanish

universities according to their journal publication profile by

areas

> Analyze the role of universities according to their centrality

values

Methodology > Unit of analysis: Spanish universities with at least 125

publication in the 2007-2011 time period

> Indicators: Production – % Q1 publications – Activity

Index – Closeness centrality

> Areas: Social Sciences – Exact Sciences – Engineering &

Technology – Life Sciences – Health Sciences

35

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Economics & Business

Social Sciences >30% Q1 journals

15-30% Q1 journals

<15% Q1 journals

36

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

High Transitivity!!

Exact Sciences >55% Q1 journals

40-55% Q1 journals

<40% Q1 journals

37

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Engineering & Technology >60% Q1 journals

35-60% Q1 journals

<35% Q1 journals

Two communities

Geographical proximity

38

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Life Sciences >60% Q1 journals

45-60% Q1 journals

<45% Q1 journals

Core universities

39

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Health Sciences >45% Q1 journals

25-45% Q1 journals

<25% Q1 journals

Cluster 1

Cluster 2

Cluster 3

40

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Characteristics Examples

Class I

•Large size

•Generalist

•Higher centrality values

Barcelona

Complutense

Autónoma de Madrid…

Class 2

•High specialization in a certain area

or research field

Politécnica de Cataluña

Navarra

Carlos III Madrid…

Class 3

•High specialization in a certain area

•Different publication profile

(outlier)

Pompeu Fabra

Class 4

•Small size

•Poor research performance

Burgos

León

Jaén…

Spanish universities according to their disciplinary focus

41

Rationale Background Research questions Contributions Conclusions

Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Conclusions

> Science maps are powerful tools for research policy makers

that complement the information provided by rankings

> Four classes of Spanish universities were found by analyzing

their journal publication profile

> Catalan universities and universities from Madrid prevail in

all areas

> Pompeu Fabra as an outlier in the Spanish university system

Rationale Background Research questions Contributions Conclusions

Table of contents

42

Background • Bibliometrics for evaluating

• National Performance research funding systems

• International Rankings

Research questions • Are national university rankings necessary in an international

context?

• How can we develop tools to identify disciplinary profiles?

Contributions • Complementing international rankings

• Mapping techniques for interpreting rankings

Conclusions • Discussion

• Implications

43

Rationale Background Research questions Contributions Conclusions

Rankings and research policy

> University rankings respond to a demand for

accountable and easy-to-use tools by research managers

> If one is to use rankings’ information to analyze

research performance they should not focus just on one

44

Rationale Background Research questions Contributions Conclusions

Are national university rankings necessary

in an international context?

> International rankings should not be used to assess

national university systems as they focus on a non-

representative sample of universities

> National rankings may complement this information by

positioning the rest of the universities in the system

> Comparisons should be made by research field in order

to draw reliable conclusions

45

Rationale Background Research questions Contributions Conclusions

The invisible factor

> There are interpretation issues on the use of rankings

by fields of difficult solution, if any

> Science maps may be helpful complementing the

information of rankings

46

Rationale Background Research questions Contributions Conclusions

How can we develop tools to identify

disciplinary profiles?

> Biplot analysis

I. Relate variables and cases

II. Analyze multivariate data

> Journal publication profile

I. Broad definition of the reasons behind the

similarity of two institutions

II. Not dependent on a classification system

47

Rationale Background Research questions Contributions Conclusions

What kind of relations are defined by the

journal publication profile?

> Geographical and historical ties

> Disciplinary relations

> Research orientation

> Publication patterns and outliers

> Size dependence ¿?

48

Rationale Background Research questions Contributions Conclusions

The analysis of disciplinary differences at the institutional level is far too complex for us to be able to reduce it to league tables

Science mapping techniques are a good way to complement the information provided by rankings as they are easy-to-read tools [..] they provide far richer information

49

‘Nothing is ever quite true’, said Lord Henry

Oscar Wilde

The Picture of Dorian Gray

Classifying and visualizing the disciplinary focus of universities

THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS

PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014

Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar