The Convergence of University Rankings and System Benchmarking

The Convergence of University Rankings and System Benchmarking

An Apparent Paradoxof “Rankology”

IREG - Warsaw, 16-17 May 2013 2

Questions

Two approaches: University RankingsSystem Benchmarking

Are they: Complementary? Competing? Consistent?


Outline

(1) Background: from ranking to benchmarking

(2) Method of investigation

(3) Results

(4) Interpretation and conclusion


(1) University Rankings


U Rankings: a Polarizing Exercise

U Rankings: hated/loved, criticized/commended,threatening/stimulating

but proliferating (“here to stay”)

Ph. Albatch’s advice [“Don’t take too much notice of rankings” (UWN, March 23, 2013)]: unlikely to be widely followed

More pitfalls discovered, uncovered, elucidated more attempts to improve methods


U Rankings: the DiseaseMethodological caveats

Biases: Research, English, STEMComposite indicators: Weighting => ElitismSubjective (reputation) /non transparent

Dangerous use (“misuses”, “abuses”)Universities: (1) Focus on competition with others instead of own

improvement / Affect strategic planning(2) Focus on biased criteria (research)

Policy makers: Focus on a few WCUs instead of whole system

Students: Impact on university selection Overall: Impact on financing

Commercialization (crowded) market


From Ranking to Benchmarking

“If Ranking is the Disease, Is Benchmarking the Cure?”

(Jamil Salmi, Sunita Kosaraju. Evaluation in Higher Education, Vol. 5 no.1, June 2011)

“Rankings: Neither a Disease nor a Cure”(Ph. Albatch, UWN, 2013)


(2) System Benchmarking

Resources

Access

Equity

TE SYSTEM

Governance

Quality control

Private ProvidersEconomic, Social

& Technological Environment


Benchmarking: Objective & Criteria

Objective: assess strength, health and performance of countries' tertiary education systems

Criteria: resources, inputs, governance, outputs and outcomes of the system (access, equity, quality, relevance)


Benchmarking: Main Initiatives

• SABER: System Approach for Better Education Results (World Bank) Still under construction

• U21 (Universitas 21/ University of Melbourne) Most recent, comprehensive available case See below

• Benchmarking University Governance (World Bank – MENA): Hybrid

• AHELO: Assessment of Higher Education Learning Outcomes (OECD) Still under experimentation


Hypothesis

Benchmarking developed in reaction to Rankings

Objectives, level of observation and criteria of Benchmarking and Ranking are quite different

==Shouldn’t they yield different results?


Method (1)

1/ Select 4 of the more popular university rankings: ARWU, THE, QS, WEBOmetrics

2/ Pick the most recent system benchmarking: U21

3/ Compare their results


Method (2)

Issue: How to compare U and Systems?

Solution: Translate U rankings into Country Rankings

Method: From: number of top universities

to: number of tertiary aged youths in one country potentially served by top universities in that country

(e.g. supply of top universities)

NB: no correlation between the 2 measures


NB: Number of Top 400 U and Supply of Top 400 U (THE) : Rank)

0

5

10

15

20

25

30

35

40

45

21

13

19

11

16

22

26

10

23

5

China

3

14

24 25

12

8

2

27

6 7

4

31

9

30

India

1820

37

32

36

17

34

29

15

28

Iceland

38 39

3335

Supply (density)

Nbr of top 400 Uni


Method (3)

Quick look at the 4 leagues selectedThe “sample”: Top 400 universities

THE ARWU QS WEBO

Nbr of countries with at least one top 400 university in each league 41 38 45 41

Nbr of countries with at least one top 400 university found in all 4 leagues (Overlap) 34 34 34 34

Nbr of top 400 universities in the countries with at least one top 400 university found in all 4 leagues (Overlap)

389 394 378 387

The 34Australia JapanAustria MexicoBelgium Netherlands

Brazil New ZealandCanada NorwayChina Poland

Czech Republic PortugalDenmark Russian FederationFinland SingaporeFrance South Africa

Germany South KoreaGreece Spain

Hong Kong SwedenIndia Switzerland

Ireland TaiwanIsrael United KingdomItaly United States


Comparing the results of the 4 Rankings (1)

Correlation between results of the 4 leagues: (Number of top universities in each country)

United Kingdom

Germany

Canada

Australia

FranceJapan

ChinaIta

ly

Netherland

Sweden

South KoreaSpain

Switzerla

nd

BelgiumIsr

aelBrazil

Hong Kong

Taiwan

Denmark

Austria

Finland

Ireland

Norway

Greece

New Zealand

Poland

Singapore

South Africa

Czech

India

Mexico

Portugal

Russia

0

10

20

30

40

50

60

Webo

THE

ARWU

QS



Correlation between results of the 4 leagues:(1) number of top universities in each country

Nbr of Top 400 Universities: R2

THE QS ARWU WEBOTHE 0.98 0.98 0.96QS 0.95 0.93ARWU 0.98WEBO



Correlation between results of the 4 leagues:(2) Supply of top universities

Density: R2

THE QS ARWU WEBO

THE 0.96 0.87 0.78

QS 0.83 0.72

ARWU 0.86

WEBO

Supply: Nbr of top U/ TE aged population

The first five countries QS ARWU THE WEBO

1 Finland 16.1 6.9 11.5 9.22 New Zealand 14.5 4.8 14.5 2.43 Switzerland 13.4 11.8 13.4 11.84 Ireland 13.3 8.0 13.3 5.35 Denmark 11.5 9.2 11.5 9.2

The last five countries QS ARWU THE WEBO

30 Poland 0.3 0.5 0.5 0.831 Mexico 0.1 0.1 0.1 0.132 Brazil 0.1 0.2 0.1 0.433 China 0.1 0.1 0.1 0.234 India 0.04 0.01 0.02 0.01


Benchmarking: “U 21”Method (1)

1/ A priori selection of 48 countries ( +2)

2/ Assessment of countries’ performance based on one overall indicator and 4 “measures”:

(1) Resources (2) Environment(3)Connectivity (4)Output


Benchmarking: Method (2)

(1) Resources (25%): 5 indicators on expenditures

(2) Environment (25%): 2 indicators on gender balance, 1 indicator on data quality, 3 indicators on policy and regulatory

environment,

1 homegrown index on internal governance


Benchmarking: Method (3)

(3) Connectivity (10%):

2 indicators on degree of internationalization (students & research)

(4) Output (40%):5 indicators on research,1 indicator on Probability of a person to attend a top 500 university

(*) based on ARWU…1 indicator on enrollment1 indicator on tertiary educated population1 indicator on unemployment among tertiary educated population


Benchmarking: Links between the 5 measures

OverallResources

(25%)Outputs

(40%)Environment

(25%)Connectivity

(25%)

Overall 0.88 0.93 0.63 0.59

Resources 0.75 0.39 0.43

Outputs 0.50 0.38

Environment 0.40

Connectivity


Comparing Results of Rankings and Benchmarking (1a)

Countries Overlap between UR and SB:

U21 & THE: 37 common countriesU21 & QS: 40 common countriesU21 & ARWU: 37 common countriesU21 & WEBO: 41 common countries

Essentially same pool of countries


Comparing Results of Rankings and Benchmarking (1b)

Not in U21Not in one (or more) Ranking

Colombia ArgentinaEstonia BulgariaIceland ChileLebanon CroatiaOman HungaryPhilippines IndonesiaSaudi Arabia IranUAE Malaysia

RomaniaSlovakiaSloveniaThailandTurkeyUkraine


Comparing Results of Rankings and Benchmarking (2)

U21 THE QS ARWU WEBO

Overall 0.74 0.73 0.76 0.76Ressources 0.69 0.69 0.78 0.77Outputs 0.55 0.57 0.63 0.62Environmment 0.50 0.49 0.45 0.38Connectivity 0.73 0.65 0.63 0.58

Correlation between U21 Indicators and Rankings (Supply): R2



U21 (Overall) and THE Rankings (R2= 0.74)

United States

Sweden

Canada

Finland

Denmark

Switzerla

nd

Norway

Australia

Netherland

United Kingdom

Singapore

Austria

Belgium

New Zealand

France

Ireland

Germany

Hong KongIsr

aelJapan

Taiwan

South Korea

Portugal

Spain

Czech

Rep

Poland

Greece Italy

RussiaChina

Brazil

Thailand

Iran

Mexico

Turkey

South Africa

India0

20

40

60

80

100

120

140

160

USA

Sweden

Switzerland New Zealand

France

Ireland

Hong KongU21

Supply

Logarithmic (Supply)



U21 (Resources) & ARWU (Supply): R2 = 0.78

U21 (Resources) ARWU (Supply)

U21 (Resources) ARWU (Supply)

Canada 100 60 South Korea 60 16Denmark 97 92 New Zealand 59 48Sweden 94 136 Portugal 58 13USA 92 46 Spain 58 23Norway 92 74 Iran 57 1Finland 89 69 UK 56 60Switzerland 87 118 Japan 53 19Singapore 82 48 Poland 49 5Netherland 80 90 Italy 47 31Austria 75 46 Czech Rep 47 12Ireland 72 80 Russia 43 1Belgium 69 72 Brazil 42 2France 67 32 Mexico 40 1Hong Kong 64 86 Hungary 40 12Israel 64 80 Argentina 39 2Germany 64 46 South Africa 35 3Taiwan 63 16 China 33 1Australia 63 82 India 23 0.1Greece 63 24


Conclusions /Interpretation

1/ Hypothesis not confirmed:a/ same set of countriesb/ similar results

2/ Two types of explanations:a/ methodologicalb/ structural


Epilogue

• System Benchmarking ends up ranking countries

• Boundaries between UR and SB are blurred• SB suffers common symptoms with UR • Convergence of the two streams of

“Rankology” not surprising• Benchmarking needs to expand its pool of

countries to become more relevant


Take Away


Thank You

SB

UR

Documents

The Convergence of University Rankings and System Benchmarking