Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings

Classifying and visualizing the disciplinary focus of universities

THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS

PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014

Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar

University of Granada

Rationale Background Research questions Contributions Conclusions

Personal motivation

The I-UGR Rankings aim to complement international rankings and account for disciplinary differences.

My contribution to this project is reflected in this thesis.

2

1th edition published in 2010


Personal motivation

3


Rankings as bibliometric tools

4

[…] rankings tend to emphasize vertical differences between institutions […]. At the same time, they obscure horizontal differences.

Marginson, 2007

[…] ‘mediocrity’ of a university does not necessarily imply that it is mediocre in all disciplines.

Moed, 2008


Rankings in research policy

5

International rankings are focused on

World-Class universities

However, many research

managers use them to

evaluate their national system


Table of contents

6

Background • Bibliometrics for evaluating

• National Performance research funding systems

• International Rankings

Research questions • Are national university rankings necessary in an international

context?

• How can we develop tools to identify disciplinary profiles?

Contributions • Complementing international rankings

• Mapping techniques for interpreting rankings

Conclusions • Discussion

• Implications

7


Bibliometrics as an evaluation tool

The need for bibliometric indicators arises from the enormous growth in scientific and technological activity that has been occuring for centuries, and has now reached avalanche proportions.

Narin et al., 1994

Type of indicator Type of

measurement Unit of analysis

Papers

Citations

Usage

Data

Altmetrics

…

Activity measurement

Impact measurement

Linkage measurement

Micro level Meso level Macro level

8


Why the need to assess universities’

performance

I. Research as the Second Mission of universities

II. The expansion of Big Science

III. Knowledge-based economies and the production

of human capital in a globalized context

IV.Economic constraints and the need for

accountability

9


National funding systems and

International Rankings

10


The problem with international rankings

11



Rankings consider universities as homogeneous,

neglecting their disciplinary differences

Health &

Life Sciences

Natural & Exact

Sciences

Social Sciences Humanities

12



Can we use international rankings to analyze national systems?

13



Thus, the country in which a university is located plays an important role in the performance of a university

Bornmann, Mutz & Daniel, 2013

Countries typically have legal and institutional rules within which their universities function

Aghion et al., 2010


Table of contents

14





context?





• Implications


15

RQ 1. Are national university rankings necessary in an international context?

• An insight into the importante of

national rankings in an international

context

• What do university rankings by fields

rank?


16

RQ 2. How can we develop tools for research policy-makers that can allow them to identify disciplinary profiles?

• What do university rankings by fields rank?

• On the use of biplot analysis

• Mapping academic institutions according to their journal publication profile

• Análisis de redes de las universidades españolas


Table of contents

17





context?





• Implications

18


An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities

Research questions > Are national university rankings necessary, considering the

existence of international rankings?

> Levels of agreement between national and international

rankings regarding their coherence and concordance

Methodology > Rankings analyzed: Shanghai Ranking – QS Ranking –

NTU Ranking – Leiden Ranking – I-UGR Rankings

> Indicators: Spearman rank correlation – Coincidence of

top Spanish universities

19


An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities

Conclusions

> Spain is poorly represented in international rankings

25% of the Spanish system

> Despite methodological differences, national rankings are

a necessary complement of international university

rankings

> We must consider these methodological differences

derived from the construction of fields as well as the indicator

used

20


What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

Research questions > Do rankings by fields represent the structure of

universities?

> Can we provide indicators that show the degree of

correspondence between each field and organizational unit?

Methodology > Universities analyzed: Pompeu Fabra – Granada

> Data processing: Identification of organizational units

> Indicators: Betweenness Centrality – Gini Index –

Number of subject – Number of disciplines

21



Results

Faculties Departments

University of Granada

22



Results

Faculties Departments

Pompeu Fabra University

23



Conclusions

> Working with addresses adopting a top down approach has

implicit problems of difficult solution

> The structure and size of universities varies significantly

from one to other which prevent from using organizational

structures to construct fields

> The Gini coefficient and the Betweenness Centrality are

useful indicators for pointing out discrepancies between fields

and organizational units

24


How can we identify disciplinary

profiles of universities?

Using mapping techniques to analyze

the multidimensional nature of

universities

25


On the use of biplot analysis for multivariate bibliometric and scientific indicators

Objective

> Present the visualization methodology known as Biplot

Analysis

Methodology > Three case studies: European countries – Universities –

Scientific fields

> Validation: Comparison with other visualization

techniques: CA – MDS – PCA

26



27



Comparison with other techniques

28



Case study 2. Universities

29



Conclusions

> Biplot analysis may well be an important analysis tool for

bibliometric studies

> They are easy to read tools of potential use in the research

policy arena

> As well as representing the relations between cases, they

also represent relations between variables, and variables and

cases

30


Mapping academic institutions according to their journal publication profile: Spanish universities as a case study

Objective

> Present a novel methodology for representing universities

according to their journal publication profile

Structure > Methodological proposal

> Case study with Spanish universities

31



Hypothesis

Universities with a similar research profile should publish in the same scientific journals

32



Information and Communication Technologies

Collaboration and geographical proximity

Basic vs. Applied

40-50% Q1 journals

30-40% Q1 journals

<30% Q1 journals

33



Conclusions

> Mapping techniques end with limitations derived from a

rigid classification system of universities

> Science maps combined with impact or visibility indicators

shows vertical stratification and horizontal diversification

> The technique highlights geographical, social and/or

historical relationships

> Limitation: Too many links between universities blur

similarities between low perfomance universities

34


Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas

Objectives > Map and analyze the research perfomance of Spanish

universities according to their journal publication profile by

areas

> Analyze the role of universities according to their centrality

values

Methodology > Unit of analysis: Spanish universities with at least 125

publication in the 2007-2011 time period

> Indicators: Production – % Q1 publications – Activity

Index – Closeness centrality

> Areas: Social Sciences – Exact Sciences – Engineering &

Technology – Life Sciences – Health Sciences

35



Economics & Business

Social Sciences >30% Q1 journals

15-30% Q1 journals

<15% Q1 journals

36



High Transitivity!!

Exact Sciences >55% Q1 journals

40-55% Q1 journals

<40% Q1 journals

37



Engineering & Technology >60% Q1 journals

35-60% Q1 journals

<35% Q1 journals

Two communities

Geographical proximity

38



Life Sciences >60% Q1 journals

45-60% Q1 journals

<45% Q1 journals

Core universities

39



Health Sciences >45% Q1 journals

25-45% Q1 journals

<25% Q1 journals

Cluster 1

Cluster 2

Cluster 3

40



Characteristics Examples

Class I

•Large size

•Generalist

•Higher centrality values

Barcelona

Complutense

Autónoma de Madrid…

Class 2

•High specialization in a certain area

or research field

Politécnica de Cataluña

Navarra

Carlos III Madrid…

Class 3

•High specialization in a certain area

•Different publication profile

(outlier)

Pompeu Fabra

Class 4

•Small size

•Poor research performance

Burgos

León

Jaén…

Spanish universities according to their disciplinary focus

41



Conclusions

> Science maps are powerful tools for research policy makers

that complement the information provided by rankings

> Four classes of Spanish universities were found by analyzing

their journal publication profile

> Catalan universities and universities from Madrid prevail in

all areas

> Pompeu Fabra as an outlier in the Spanish university system


Table of contents

42





context?





• Implications

43


Rankings and research policy

> University rankings respond to a demand for

accountable and easy-to-use tools by research managers

> If one is to use rankings’ information to analyze

research performance they should not focus just on one

44


Are national university rankings necessary

in an international context?

> International rankings should not be used to assess

national university systems as they focus on a non-

representative sample of universities

> National rankings may complement this information by

positioning the rest of the universities in the system

> Comparisons should be made by research field in order

to draw reliable conclusions

45


The invisible factor

> There are interpretation issues on the use of rankings

by fields of difficult solution, if any

> Science maps may be helpful complementing the

information of rankings

46


How can we develop tools to identify

disciplinary profiles?

> Biplot analysis

I. Relate variables and cases

II. Analyze multivariate data

> Journal publication profile

I. Broad definition of the reasons behind the

similarity of two institutions

II. Not dependent on a classification system

47


What kind of relations are defined by the

journal publication profile?

> Geographical and historical ties

> Disciplinary relations

> Research orientation

> Publication patterns and outliers

> Size dependence ¿?

48


The analysis of disciplinary differences at the institutional level is far too complex for us to be able to reduce it to league tables

Science mapping techniques are a good way to complement the information provided by rankings as they are easy-to-read tools [..] they provide far richer information

49

‘Nothing is ever quite true’, said Lord Henry

Oscar Wilde

The Picture of Dorian Gray

Classifying and visualizing the disciplinary focus of universities

THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS

PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014

Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar

Education

Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings