1

Academic Ranking of World UniversitiesMethodologies and Problems

Academic Ranking of World UniversitiesMethodologies and Problems

August, 2008August, 2008

By Professor Nian Cai LIUBy Professor Nian Cai LIU

Center for World-Class Universities, Graduate School of EducationCenter for World-Class Universities, Graduate School of Education

Shanghai Jiao Tong University, ChinaShanghai Jiao Tong University, China

2

Outline

Purposes1

Methodologies & Results2

Problems & Discussion33

Ranking by Subject Fields44

35 Final Remarks5

3

Purposes Purposes Purposes Purposes

4

Dream of Chinese for WCU

World-class university (WCU) is a dream for

generations of Chinese. It’s not only for pride, but also

for the future of China.

Recently, Chinese government has launched several

initiatives for research universities. The best-known one

is specially designed to build WCU (985 Project).

5

Goals of Top Chinese Universities

Many top Chinese universities have setup their

strategic goals as WCU.

Most of them have also set time tables for achieving the

goal of WCU. For example:

2016 for Peking University

2020 for Tsinghua University

6

Questions About WCU

Is there a clear definition for WCU?

How many WCU should there be in the world?

What are the positions of top Chinese universities in

the world?

How can Chinese universities improve themselves to

reach the goal of WCU?

7

Academic Ranking of World Universities

Our original purpose of doing the Academic Ranking of

World Universities (ARWU) was to find out the position

of Chinese universities in the world and the gap between

them and WCU.

ARWU was put on the internet upon the encouragement

of colleagues from all over the world.

8

Features of ARWU

ARWU uses a few carefully selected, objective criteria and

internationally comparable data that everyone could

verify in some way.

It has been carried out by a ranking team of four

researchers at Shanghai Jiao Tong University for their

academic interests.

It has been done independently without any financial

support from any external sources.

9

Methodologies Methodologies & Results& Results

Methodologies Methodologies & Results& Results

10

Selection of Universities

Any university that has any Nobel Laureates, Fields

Medals, Highly Cited Researchers, or papers published

in Nature or Science.

Major universities of every country with significant

amount of papers indexed by Thomson.

Number of universities scanned: >2000

Number of universities actually ranked: >1000

Number of ranked universities on our web: 500

11

Ranking Criteria and Weights

12

Definition of Indicator: Alumni

The total number of the alumni of an institution winning Nobel Prizes and Fields Medals.

Alumni are defined as those who obtain bachelor, Master’s or doctoral degrees from the institution.

Different weights are set according to the periods of obtaining degrees. The weight is 100% for alumni of 1991-2000, 90% for alumni of 1981-1990, 80% for alumni of 1971-1980, and so on.

If a person obtains more than one degrees from an institution, the institution is considered once only.

13

Definition of Indicator: Award The total number of the staff of an institution winning Nobel

prizes in physics, chemistry, medicine and economics and Fields Medal in Mathematics.

Staff is defined as those who work at an institution at the time of winning the prize.

Different weights are set according to the periods of winning the prizes. The weight is 100% for winners since 2001, 90% for winners in 1991-2000, 80% for winners in 1981-1990, 70% for winners in 1971-1980, and so on.

If a winner is affiliated with more than one institution, each institution is assigned the reciprocal of the number of institutions.

For Nobel prizes, if a prize is shared by more than one person,

weights are set for winners according to their proportion of prize.

14

Definition of Indicator: HiCi

The number of highly cited researchers in 21 broad subject categories in life sciences, medicine, physical sciences, engineering and social sciences.

The definition of categories and detailed procedures can be found at the website of Institute of Scientific Information.

The total number of HiCi is about 5000, about 4000 of which is university staff.

15

Definition of Indicator: N&S

The annual average number of articles published in Nature and Science in the past five years.

To distinguish the order of author affiliation, a weight of 100% is assigned for corresponding author, 50% for first author (second author if the first author is the same as corresponding author), 25% for the next author, and 10% for other authors.

Only publications of article type are considered.

16

Definition of Indicator: PUB

Total number of articles indexed in Science Citation

Index-expanded (SCIE) and Social Science Citation

Index (SSCI) in the past year.

A weight of 2 is assigned to articles indexed in SSCI

to compensate the bias against humanities and social

sciences.

Only publications of article type are considered.

17

Definition of Indicator: PCP

The sub-total scores of the above five indicators divided by the number of full-time equivalent academic staff.

If the number of academic staff for institutions of a country cannot be obtained, the total scores of the above five indicators is used.

For ranking 2005, the number of full-time equivalent academic staff is obtained for institutions in USA, China, Australia, Italy, Netherlands, Sweden, and Belgium etc.

18

Main Sources of Data

Nobel laureates:

http://www.nobel.se

Fields Medals:

http://www.mathunion.org/medals/

Highly-cited researchers:

http://www.isihighlycited.com

Articles published in Nature and Science:

http://www.isiknowledge.com

Articles indexed in SCIE and SSCI:

http://www.isiknowledge.com

19

Results of ARWU

http://www.arwu.org/

Top 500 universities in the world

Top 100 universities in North and Latin America

Top 100 universities in Asia/Oceania

Top 100 universities in Europe

Statistics of top universities by region and country

Percentage distribution of top universities by country as compared with the share of global population and GDP

20

ProblemsProblems & &

DiscussionDiscussionProblemsProblems & &

DiscussionDiscussion

21

Methodological: Education and Service

Education is the basic function of any university, however, it would be impossible to rank the quality of education due to the huge differences among the national systems.

Contribution to the national economic development is becoming increasingly important for universities, however, it is impossible to obtain internationally comparable indicators and data.

The academic or research performance of universities, a good indication of their reputation, can be ranked internationally.

22

Methodological: Humanities & Social Sciences

Many well-known institutions specialized in humanities and social sciences are ranked relatively low.

Since 2004, the indicator of N&S is not considered for institutions specialized in humanities and social sciences, its weight is relocated to other indicators.

Since 2005, a weight of 2 for articles indexed by SSCI is considered.

Nevertheless, if a university specialized in social sciences and humanities had Nobel Laureates in economics and Highly Cited Researchers in social sciences, it should have good standing.

23

Methodological: Language Bias

English is the language of international academic community.

Any ranking based on academic performance will be biased towards institutions in English-speaking countries.

One possible solution: papers published in non-native languages are offered a special weight.

Another possible solution: normalization of total articles by the proportion of journal editors of each country.

24

Methodological: Award and Alumni

Universities which started after 1911 do not have a fair chance.

Disciplines not related to the awarding fields do not have a fair chance. Other important awards include Abel, Pulitzer, Turing, Tyler, Pritzker, etc.

Institutions for winning awards and those for doing the researches may not be the same.

Institutions for obtaining degrees and those for pursuing the studies may not be the same.

Postdoctoral training is not considered.

25

Methodological: Per Capita Performance

The weight of the PCP indicator for per capita performance is rather low. Large institutions have relatively high positions in the ranking.

However, it’s very difficult to obtain internationally comparable data on the number of academic staff.

The types of academic staff: such as purely teaching staff, teaching and research staff, purely research staff.

The ranks of academic staff: such as professor, associate professor, reader, lecturer, research scientist etc.

26

Technical: Attributions

Many universities have more than one commonly used names: such as Virginia Tech and Virginia Polytechnic and State University.

Variations due to translation: such as Univ Koln and Univ Cologne, Univ Vienna and Univ Wien.

Abbreviated names: such as ETH Zurich for Swiss Federal Institute of Technology Zurich.

Some authors only write their departmental or institute names without mentioning their university names.

27

Technical: Definition of Institution

University systems: such as Univ California system, Univ London system.

Affiliated institutions and research organizations: such as Ecole Polytechnique Montreal (affiliated to University of Montreal), CNRS Labs (affiliated to French universities).

Teaching and affiliated Hospitals: complex!

Our answer: according to author’s expression.

28

Other Technical Problems

Merging, splitting, inheriting, discontinuing, name-changing of institutions such as:

Univ Kwazulu-Natal in South Africa merged from Univ Natal and Univ Durban-Westville.

University of Innsbruck in Austria splitted into Univ Innsbruck and Innsbruck Medical Univ.

Humboldt Univ Berlin and Free Univ Berlin inheriting the Nobel Prizes of the Berlin University before world war II.

29

Ranking byRanking by Broad Subject FieldsBroad Subject Fields

(ARWU-FIELD)(ARWU-FIELD)

Ranking byRanking by Broad Subject FieldsBroad Subject Fields

(ARWU-FIELD)(ARWU-FIELD)

30

Requests for Ranking of World Universities by

Broad subject fields or schools, colleges

and

Subject fields or programs, departments

★ In addition, many top Chinese universities want to

learn their positions in the world by broad subject

fields or disciplines.

31

Definition of Broad Subject Fields

Natural Sciences and Mathematics (SCI)

Engineering/Technology and Computer Sciences (ENG)

Life and Agriculture Sciences (LIFE)

Clinical Medicine and Pharmacy (MED)

Social Sciences (SOC)

Arts and humanities are not ranked

32

ARWU-FIELD Indicators and Weights

Code SCI ENG LIFE MED SOC

Alumni 10% 10% 10% 10%

Award 15% 15% 15% 15%

HiCi 25% 25% 25% 25% 25%

TOP 25% 25% 25% 25% 25%

PUB 25% 25% 25% 25% 25%

Fund 25%

33

Changes in Indicators and Definition

N&S in ARWU is not used in ARWU-FIELD.

TOP is the percentage of articles published in the top

20% journals of each broad subject field.

Fund is the total engineering-related research

expenditures. It’s used only for ENG ranking.

Alumni and Award since 1951 are used for all rankings

fields except ENG.

PUB is the total number of articles indexed by Thomson

in the past year.

34

Results of ARWU-FIELD http://ed.sjtu.edu.cn/ranking.htm

Top 100 universities in SCI

Top 100 universities in ENG

Top 100 universities in LIFE

Top 100 universities in MED

Top 100 universities in SOC

Statistics of top universities by region & country

List of top universities by number of top fields

35

Special Problems in ARWU-FIELD

It’s difficult to obtain data on engineering-related

research expenditures and make them comparable.

For 2007, Fund was obtained only for US and

Canadian universities.

It’s difficult to separate the Nobel Laureates in

Physiology or Medicine. They are used in both LIFE

and MED ranking.

36

Final RemarksFinal RemarksFinal RemarksFinal Remarks

37

Controversy of Ranking

Any ranking is controversial and no ranking is

absolutely objective.

University rankings become popular in many

countries. Whether we agree or not, ranking systems

clearly are here to stay.

The key issue then becomes how to improve ranking

systems for the benefits of higher education (IREG).

38

Use of Ranking

Rankings should be used with cautions. Their

methodologies must be read carefully before

reporting or using their results.

Rankings should be used in combination with other

types of evaluation whenever possible, such as

bench-marking, peer review, etc.

39

Future Efforts

Study all the above mentioned problems and

continuously improve the ranking methodologies.

Establish more comprehensive databases of

WCU.

Update ARWU (every August) and ARWU-

FIELD annually (every February).

40

Future Efforts (Cont’ed)

Provide ranking of universities specialized or

strong in engineering, medicine, etc. based on the

classification of world universities.

Provide ranking of universities with different

size, history, budget and function etc. once

internationally comparable data are obtained.

41

http://gse.sjtu.edu.cn/

http://www.arwu.org