University and Higher Education Rankings – What Relevance Do they Have?

EI Affiliates Conference in the OECD member countries

“FRAMING EDUCATION FOR THE PUBLIC GOOD”

29-30 January 2013, London, United Kingdom

Grahame McCulloch

General Secretary, NTEU (Australia)

EI Executive Board Member

BACKGROUND AND CONTEXT• Rise of mass higher education in rich countries

(1960s-1990s) – mainly well funded public systems (with notable Japanese and Korean exceptions) but recognisable hierarchies and stratification (with a premium on research intensity)

• Economic, social and labour market benefits (including R&D, innovation, technology transfer and human capital)

BACKGROUND AND CONTEXT

• Slowing of growth and scarcer resources – increased managerial authority, expanded role for private effort and markets, accountability, performance measurement and indicators and erosion of tenure and academic autonomy (1990s – present)

• Rapid growth in emerging and developing countries (embryo of mass systems) and strong preoccupation with science, technology, R&D and direct economic role of universities (particularly in Asia)

BACKGROUND AND CONTEXT

• 177 million students, 17000 institutions and 11 million staff – expected to double in 20 years (mainly in Asia and Latin America)

• Global trade and flows – 2.5 million international students (around $100 billion), regional trade blocs (US, Canada, UK, Australia and Europe) and regional/national accreditation, qualifications and quality assurance

BACKGROUND AND CONTEXT

• Imbalance between per capita resources and enrolments on world scale

• Rich countries seeking larger share of world expansion via trade, offshore and joint ventures and research collaboration

• Emerging countries seeking domestic expansion through national strategy and investment (including imports of foreign capital, expertise and technical systems)

Source: UNESCO 2010

Source: UNESCO 2010

Chile

Korea

Argen

tina

Slova

k Rep

ublic

Poland

United K

ingdom

Estonia

Mex

ico

Japan

Hungary

Portugal

Isra

el

Czech

Rep

ublicIta

ly

New Z

eala

nd

Slove

nia

Austra

lia

OECD ave

rage

Icel

and

EU21 a

vera

ge

United S

tate

s

Spain

France

Nether

lands

Austria

Belgiu

m

Finla

nd

Denm

ark

Sweden

Norway

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,0001,

501 2,

751

3,30

9 4,50

0

4,53

5

4,64

4

4,81

8

5,50

9

6,10

2

6,25

3

6,37

2

6,43

1

6,45

5

6,48

3

7,21

2

7,31

2

7,49

6 8,81

0

9,14

4

9,61

3

9,67

9 10,6

16

12,1

89

12,2

52

12,5

00 13,5

24

15,8

68 17,2

52

17,8

48 18,6

23

Annual Public Expenditure per Student on Tertiary Education In-stitutions

($US PPP 2009 )

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

Philippines Fiji Thailand Malaysia New Zealand

Australia Japan United Kingdom

Hong Kong United States

Denmark

Public Expenditure per Pupil in Tertiary Education ($US PPP)2005

Source: NTEU Estimates derived from:UNESCO (Public Expenditure per Tertiary Pupil as % of GDP per Capita) and Mundi Index GDP per Capita ($US PPP) (http://indexmundi.com)

OVERVIEW OF GLOBAL MEASUREMENT INSTRUMENTS

• Parallels between schools, vocational education and higher education – Program for International Student Assessment (PISA), Teaching and Learning Survey (TALIS), Program for International Assessment of Adult Competencies (PIAAC) and Assessment of Higher Education Learning Outcomes (AHELO)

• Focussed on systems and/or disciplines (not individual institutions) for cross-country comparisons and use in national benchmarking and quality assurance processes, high media and political visibility

OVERVIEW OF GLOBAL MEASUREMENT INSTRUMENTS

• Problems of measurement and misinterpretation or misuse of data by national governments – causation, correlation, limits of mathematical language and dangers of simplistic international league tables and single standardised scores

• Narrowing of domestic public policy standards with strong instrumental focus, and less emphasis on wider social and educational objectives.

OVERVIEW OF GLOBAL MEASUREMENT INSTRUMENTS

• Tendency of scores and metrics to undermine qualitative and organisational quality assurance measures, and to encourage ‘gaming’ and manipulation of metrics

UNIVERSITY RANKING SYSTEMS

• Siblings and first cousins of international and national performance indicators/accountability systems, but focussed on individual institutions and not systems

• Typical weighted indicators include undergraduate and postgraduate enrolments, research grants and endowments, public and private funding, student/staff ratios, graduation rates, research citations and publications and prizes/awards

UNIVERSITY RANKING SYSTEMS

• Measurement of teaching and research quality uses proxies (metrics and reputational surveys), and league tables are based on standardisation, aggregation into single score and ordinal scale based on the top ranked institutions

• Developed and administered by media companies or specialist arm of university research centres – no direct government or intergovernmental involvement

UNIVERSITY RANKING SYSTEMS

• Multi-Ranking without league tables – University Ranking and U-Map – the “Berlin Rankings” (CHE/die Zeit, Germany and IREG) and U-Multirank (EU)

UNIVERSITY RANKING SYSTEMS

• National ranking league tables – Japan (Asahi Shimbun), Canada (Macleans), Italy (La Repubblica), US (US News and World Report)

• International ranking league tables – US News and World Report (with QS Symonds), Times Higher Education Supplement (with Thomson Reuters), Academic Rank of World Universities (Shanghai Jiao Tong University, China), Global Universities Rankings (Lomonosov State University, Russia), Scientific Papers for World Universities (Accreditation and Evaluation Council, Taiwan), Leiden Research Ranking (Leiden University, Netherlands), University Web Ranking (CSIC Cybernetics, Spain)

MOST INFLUENTIAL – THES AND ARWU

• In rich countries used by governments in domestic policy debate and by universities in marketing and promotion, particularly in North and South East Asia

• In emerging and developing countries used by governments as benchmark for development of domestic institutions and systems

• Directly affects institutional behaviour and indirectly high achieving student choice

MOST INFLUENTIAL – THES AND ARWU

• ARWU based solely on metrics with research (maths and science in particular), accounting for 90% of composite scores

• THES apparently more balanced (30% teaching, 30% research volume, income and reputation, 32.5% research citations, 7.5% international and 2.5% economic innovation), but actually closer to 75% weighting for research

MOST INFLUENTIAL – THES AND ARWU• Both rankings actually reflect the prestige, high

selectivity in student enrolments and staff appointments, economic resources and global reach of each university

• Are not able and do not aspire to reflect diversity of institutions and systems (large and small, teaching intensity, access and equality, three and four year programs, cultural context)

• Not a guide or benchmark for national system development

Proportion of universities covered by THES and ARWU rankings

Source: European Universities Association (EUA) 2011

1 California Institute of Technology USA 1 Harvard University USA2 University of Oxford UK 2 Stanford University USA

2 Stanford University USA 3Massachusetts Institute of Technology

(MIT)USA

4 Harvard University USA 4 University of California, Berkeley USA5 Massachusetts Institute of Technology USA 5 University of Cambridge UK6 Princeton University USA 6 California Institute of Technology USA7 University of Cambridge UK 7 Princeton University USA8 Imperial College London UK 8 Columbia University USA9 University of California, Berkeley USA 9 University of Chicago USA10 University of Chicago USA 10 University of Oxford UK11 Yale University USA 11 Yale University USA

12ETH Zürich – Swiss Federal Institute of

Technology ZürichSwitz 12 University of California, Los Angeles USA

13 University of California, Los Angeles USA 13 Cornell University USA14 Columbia University USA 14 University of Pennsylvania USA15 University of Pennsylvania USA 15 University of California, San Diego USA16 Johns Hopkins University USA 16 University of Washington USA17 University College London UK 17 The Johns Hopkins University USA18 Cornell University USA 18 University of California, San Francisco USA19 Northwestern University USA 19 University of Wisconsin - Madison USA20 University of Michigan USA 20 The University of Tokyo Jap21 University of Toronto Can 21 University College London UK22 Carnegie Mellon University USA 22 University of Michigan - Ann Arbor USA

23 Duke University USA 23Swiss Federal Institute of Technology

ZurichSwitz

24 University of Washington USA 24The Imperial College of Science,

Technology and MedicineUK

25 University of Texas at Austin USA 25 University of Illinois at Urbana-Champaign USA

25 Georgia Institute of Technology USA 26 Kyoto University Jap27 University of Tokyo Jap 27 New York University USA28 University of Melbourne Aust 27 University of Toronto Can29 National University of Singapore Sing 29 University of Minnesota, Twin Cities USA30 University of British Columbia Can 30 Northwestern University USA

Times Higher Education 2012 Academic Ranking of World Universities

Time Higher Education Top 400 by RegionRegion No. %North America 133 33.3%

Europe 180 45.0%

Asia 80 20.0%

Africa 4 1.0%

Latin America 3 0.8%

ARWU Top 400 by Country/RegionRegion / Country No. %USA 139 34.8%Canada 19 4.8%

UK 33 8.3%Europe 128 32.0%

Asia 68 17.0%

Africa 5 1.3%

Latin America 8 2.0%

AN EI RESPONSE

• High quality information and feedback for national and international students necessary in mass systems, and robust quality assurance is essential

• Quality assurance and performance assessment should reflect the characteristics, resources social/educational objectives of each institution, and be autonomously determined within each university using peer review and stakeholder consultation

AN EI RESPONSE• Academic freedom, collegial decision-making, trade union

rights and employment standards should be part of quality assurance criteria

• The aggregation of data at national and international level for any cross-institutional comparative purposes should prevent the construction of league tables

• Building on EI’s strategic response to PISA, EI should continue a critical dialogue with OECD in the development and implementation of AHELO (noting its discipline and national system focus). Any final methodology should prevent the construction of arbitrary league tables

AN EI RESPONSE

• EI should develop direct dialogue with the Berlin rankings group (CHE/die Ziet and IREG) on the development of University Ranking and U-Map, and EU on U-Multirank (noting these are consciously constructed to enable comparison without league tables)

FURTHER READING

• Global university rankings: where to from here?, Simon Marginson, Centre for the Study of Higher Education, University of Melbourne, Australia

• To Rank or To Be Ranked: The Impact of Global Rankings in Higher Education, Simon Marginson and Marijk van der Wende, Journal of Studies in International Education, Vol. 11 No 3/4

• College and University Ranking Systems - Global Perspectives and American Challenges, Institute for Higher Education Policy, Washington D.C., April 2007

• Global University Rankings and Their Impact, EUA Report on Rankings 2011, Andrejs Rauhvargers

• The Road to Academic Excellence – The Making of World-Class Research Universities, Philip G. Altbach and Jamil Salmi Editors, The World Bank, Washington D.C