The European Commission’s scienceand knowledge service

Joint Research Centre

Tools for developing and auditing composite index frameworks and

scoreboardsDániel Vértesy, PhD

Research Fellow | European Commission, Joint Research Centre, Competence Centre on Composite Indicators and Scoreboards (CC-COIN), Ispra, Italy

UN Expert Group Meeting on Methodologies of Regional Integration Indexes 26/09/2018, Addis Ababa (ET)

3

1 IntroductionThe JRC’s work on Composite Indicators and Scoreboards

5

• information explosion in terms of the 3V’s (Volume, Variety, Velocity), but

• human ability to analyse information will increase only marginally

± 2 is the average number of objects an average human can hold in working memory

[Miller's Law]

Data Revolution

7

6

Making sense of Data, Information & Knowledge

• Diffusion of evidence-based policies

• Today’s information deluge

• Fundamental role for the JRC role in managing knowledge in support of EU policy making

7

Knowledge Centres

Disaster Risk Management Knowledge Centre

Knowledge Centre for Territorial Policies

Knowledge Centre on Migration and Demography

Knowledge Centre on Bioeconomy

Knowledge Centre for Food Fraud and Quality

Provide relevant dataknowledge intelligence

in a specific policy field

8

Competence Centres

Competence Centre on Composite Indicators and Scoreboards

Competence Centre on Text Mining and Analysis

Competence Centre on Microeconomic Evaluation

Competence Centre on Modelling

Competence Centre on Foresight

Provide scientific expertisestate-of-art methods tools

to inform policies in a any policy field

9

Competence Centre COIN

Provide

scientific expertisestate-of-art methodstools

to inform policies in any policy field

The European Commission’s Competence Centre on Composite Indicators and Scoreboards

Our aimsServe as a referencepoint for EU policymakers on monitoringmulti-dimensionalphenomena

Maintain the globalrecognition in auditingcomposite indicatorsand scoreboards

10

Competence Centre COIN

What we doSupport to EU Services (> 40 indices & scoreboards)E.g., Social Scoreboard for the European Pillar of Social Rights (26/04/2017; CAB Juncker), Cultural and Creative Cities Monitor (06/07/2017; Commissioner Navracsics)

Collaborations with international partners (> 100 audits)E.g., Global Innovation Index (INSEAD, WIPO), Commitment to Reducing Inequality Index (Oxfam), Financial Secrecy Index (Tax Justice Network)

Methodology Jointly with the OECD quality control framework (conceptual coherence, multivariate analysis, sensitivity analysis, multi-criteria decision analysis, statistics and policy)

Training & Community of Practice on Composite Indicators and Scoreboards 5-9 November 2018, Ispra, Italy (16th Training – and 3rd CoP)

11

Community of Practice on Composite Indicators and Scoreboards

Discussion Topics• Fairness, Inequality and Poverty• Social Rights & Social Trust• Resilience & Risk Management• Sustainable Development & Sustainable

Governance

• Education & Skills• Ageing societies & Migration• Innovation & Digital Transformation• Competitiveness & Attractiveness

▶ How should indices and scoreboards be used for policy and what should be avoided?▶ How can indices and scoreboards be communicated to the wider public?

Ispra, IT8-9 Nov 2018

12

[COIN Data Hub, inventory in preparation]

150•Banks’ contribution to EU Single Resolution FundDG FISMA

•Index for Risk Management (INFORM)DG ECHO

•Consumer Conditions & MarketsDG JUST

•Europe 2020 IndexDG REGIO

•Small Business Act Principles DG GROW

•Innovation Output IndicatorDG RTD

•Cultural and Creative Cities Monitor DG JRC

•ASEM Sustainable Connectivity Index (Oct. 2018)EEAS

•Systemic Risk Heat MapsECB

composite indicators and scoreboards

developed or used by the EC

50% of the EC indices/scoreboards are developed

with JRC contribution

Ever increasing number of monitoring tools

13

Example 1: Cultural and Creative Cities Monitor

https://composite-indicators.jrc.ec.europa.eu/cultural-creative-cities-monitorhttps://composite-indicators.jrc.ec.europa.eu/cultural-creative-cities-monitor

14

Example 1: Cultural and Creative Cities Monitor

15

Example 2: JRC support to the European Central Bank

ECB – Systemic Risk Country Heat MapL L L L L L L L L L L L L L L L L L L L L L L L L LL L L L L L L . L L L L L L L L L L L L L

L L L H L L L L L H L L H LH H H H H H H H H H H H H H

H H H H H H H . H H H H H H H HH H H H H H H H H H . H H H H H H H H H H H H

H H H H H H . H H H HH H H H H H H . H H H HH H H H H H H H H H . H H H H H H H H H

. . . . H H . . . . . . . . . H . .H H H . H H H H H H

L H L L . L L H L H L H L HH H H H H H H H H . H H H H H H H H H H

L L L L . L L L L L L L L L L L LH H H H . H H H H .

H H H H H H H H H HH H H H H H H H H H H H H H

H H H H H H . H H H H H H .

L L L L L L L L L L L L L L L L L L L L L L L L L LL L L L L L L L L L L L L L L L L L . L L

L L L H L L H L L H L L L LH H H H H H H H H H H H H H

H H H H H H H H H H H H . H H HH H H H H H H H H H H H H H H H H H H . H H H

H . H H H H H H H H HH H H H H H H H H . H H

H H H H H H H H H H H H H H H H H . H H. . . . . . . . H . H . H . . . . .

H H H H H H H H . HH L H H H H L L L L . L L L

H H H H H H H H H H H H H H H H . H H HL L L L L L L L L L L L L . L L L

. H H H H H . H H HH H H H H H H H H H

H H H H H H H H H H H H H HH H H H . H H H H H . H H H

Low systemic risk ------------------------------> High systemic risk

16

A compass towards better living and working conditions in the EU

• 3 dimensions & 12 areas• 14 headline & 28 secondary indicators• 93 indicators in total (gender/age)

https://composite-indicators.jrc.ec.europa.eu/social-scoreboard/

Example 3: Social Scoreboard for the European Pillar of Social Rights

17

2 Developingcomposite index frameworks:What, why, how to measure?

18

Step 10. Presentation & dissemination

Step 9. Association with other variables

Step 8. Back to the indicators

Step 7. Robustness & sensitivity

Step 6. Weighting & aggregation

Step 5. Normalisation of data

Step 4. Multivariate analysis

Step 3. Data treatment

Step 2. Selection of indicators

Step 1. Developing the framework

Finally endorsed (after 2 rounds of consultation) by the OECD high level statistical committee in March 2008

The 10 Steps Guide for Composite Indicators

19

Developing an index...

A technical challengeA technical challenge A conceptual challengeA conceptual challenge

Indicators as neutral metrics

=> Descriptive

Trigger action by comparison

=> normative

20

The power of numbers

Numbers as targets: strong advocacy tool- See the case of the Millennium Development Goals (MDGs)

“While quantification is the key strength of global goals, it alsoinvolves simplification, reification and abstraction, which havefar-reaching implications for redefining priorities”Setting MDG goals/targets influenced policy priorities and hadnormative effects on development discourses;All MDG goals/targets “led to unintended consequences indiverting attention from other important objectives and reshapingdevelopment thinking”

(Fukuda-Parr, S., Yamin, A.E., Greenstein, J., 2014)

21

Indicators for Policy

Surge of indicators of policy• Internationalization & globalization (need for comparison)• Calls for accountability (the “audit explosion” – Power, 1994)

• legitimize interventions (ex ante) & demonstrate success (ex post)

• Indicators offer de-personalized accounts on the state of the world to allow for rational decisions to change the world for the better (Rottenburg and Merry, 2015)

=> Aim to provide unbiased, comparable information on a single phenomenon of interest for different units

BUT: quantification involves simplification=> Normative choices

22

Indicators are shaped by policy needs & discourseIndicators, in turn, influence policy discourse

Indicators are embedded in a socio-political context• Indicators & indicator frameworks are value laden; reflect policy discourse

(Godin, 2002)

• Indicator developers & users better be aware of consequences:• The “agora model” (Barré, 2001, 2010)

indicators are “debating devices” – it’s the process that matters!• Honest broker (Pielke, 2007)• “Indicators in the wild” – open up scope of data & expertise, process, framing

of problems (Rafols, 2017)

23

Composite Indicators

Advantages:

• Support decision makers by summarizing complex or multi-dimensional issues

• Provide the “big picture”, highlight common trends

• Measure a latent phenomenon that is not directly measureable

• Attract public interest by benchmarking

Pitfalls:

• Offer misleading, non-robust policy messages if they are poorly constructed or misinterpreted

• May invite politicians to draw simplistic policy conclusions

• Easier to “manipulate” than individual indicators; the selection of sub-indicators and weights could be the target of political challenge

Important to ensure their qualityImportant to ensure their quality

The development process is crucial: it helps• Better understand how a system functions• Identify latent dimensions, redundancies or

trade-offs between components

The development process is crucial: it helps• Better understand how a system functions• Identify latent dimensions, redundancies or

trade-offs between components

24

The Quality of Composite Indicators

“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)

Advocacy

Analysis

Quality

Codified and continuously refined methodology• The OECD-JRC Handbook (JRC-OECD, 2008)• Audits – robustness and sensitivity

analyses (i.e. Saisana et al, 2011; Paruolo et al, 2013)

25

3 Auditingcomposite index frameworks

26

What is the purpose of auditing indicators?

• Test for statistical coherence• Are the conceptual and statistical frameworks aligned?• Is the structure statistically justified? • Are all indicators sufficiently driving index scores?

(accounting for information loss)

• Test for robustness and sensitivity • How do modelling assumptions influence results?

27

100

• Human Development Index UNDP

• 3 child-poverty indicesUNICEF

• Multidimensional Poverty Assessment ToolUN IFAD

• Global Innovation Index WIPO

• ICT Development IndexITU

• Financial Secrecy IndexTJN

requests for JRC audits by international partners (UN, UNICEF, Tax Network, WEF, INSEAD, WIPO, UN-IFAD, FAO, Harvard U., Yale U., Columbia U., Cornell U.)

The EC JRC is an in-house science centre providing technical advice to a variety of projects throughout Europe. As part of that mission it is also an expert reviewer of project methodologies, says SiddharthBannerjee, Nesta project lead for the EDCI. It’s important to be rigorous about methodology. It’s how bias can sneak in when not expected,” he adds. “The EC JRC may not be well known outside certain circles, but it is the ultimate indicator of what a composite index looks like.”

Examples: Requests for JRC audits

28

• Assess if few indicators dominate the framework

(rindicator,index>0.95)

• Assess if indicators behave as “noise” in the framework

(-0.3<rindicator,index<0.3)

• Assess if indicators are negativelyrelated to the index

(rindicator,index<-0.3)

Statistical coherence

Corr. rSens itivity Indices (r^2) Interpretation

FRAMEWORK CONDITIONSHum an resources1.1.1 New doctorate graduates 0.691 0.477 driver1.1.2 Population completed tertiary education 0.561 0.315 w eak driver1.1.3 Lifelong learning 0.844 0.713 Main driverAttractive research system s1.2.1 International sc ientif ic co-publications 0.889 0.791 Main driver1.2.2 Scientif ic publications among top 10% most cited 0.898 0.806 Main driver1.2.3 Foreign doctorate students 0.831 0.691 driverInnovation-friendly environment1.3.1 Broadband penetration 0.508 0.258 w eak driver1.3.2 Opportunity-driven entrepreneurship 0.732 0.535 driverINVESTMENTSFinance and support2.1.1 R&D expenditure in the public sector 0.731 0.534 driver2.1.2 Venture capital investments 0.507 0.257 w eak driverFirm investments2.2.1 R&D expenditure in the business sector 0.809 0.655 driver2.2.2 Non-R&D innovation expenditure -0.065 0.004 silent2.2.3 Enterprises providing ICT training 0.777 0.603 driver

Indicators vs. SII scores

SII 2017: Corr. Indicators vs. Aggregate Index

29

The correlation table says it all…

The European Innovation Scoreboard 2017The European Innovation Scoreboard 2017

30 JRC-COIN © | Step 7: Statistical Coherence

• Ideally, an indicator correlates more with its own dimension than with others

• See e.g., Sustainable Society Index, 2012

Are the indicators in the right dimension?

31 JRC-COIN © | Step 7: Statistical Coherence

• Similarly, sub-pillars are ideally relatively more correlated to their own pillar than others

• See e.g. Global Innovation Index, 2017

• Positive, strong correlation between Sub-pillar and pillar

Are the indicators in the right dimension?

32

• …• Assess if indicators statistically fit

better under different dimensions than those in the framework

• Assess if dimensions should be merged or split

• Assess if bias has been introduced in the index (e.g., due to population size, population density, GDP)

Statistical coherence

33

• Identify the sources of uncertainty in the index development

• Assess the impact of the uncertainties to the index scores or ranks

complement scores/ranks with confidence intervals

Robustness & Sensitivity

Including/excluding variables Normalisation

Missing dataWeights

Aggregation

Country 1

10

20

30

40

50

60

Country 2 Country 3

34

• Identify the sources of uncertainty in the index development

• Assess the impact of the uncertainties to the index scores or ranks

complement scores/ranks with confidence intervals

Robustness & Sensitivity

35

How important are underlying assumptions?• Identify key modelling choices• Test their impact on the final composite

scores and rankings

Assumptions influence outcome

(Source: Hardeman and Vertesy, 2015)

36

• Identify which uncertainties are more crucial in determining the final classification

• Explain why certain countries notably improve or deteriorate their relative position given the assumptions

• Robustness ≠ Quality

Robustness & Sensitivity

37

COIN helps to ensure that composite indicators are …

developed sensibly and used responsibly.

‘Sensible development’ of a composite indicator implies a quality control process based on both conceptual and statistical considerations.

‘Responsible use’ calls for care in drawing conclusions and recommendations without taking into account the conceptual context in which composite indicators were developed.

Indices are powerful advocacy tools

ec.europa.eu/jrc

@EU_ScienceHub

EU Science Hub - Joint Research Centre

Joint Research Centre

EU Science Hub

Stay in touch!

ThanksQuestions?You can find me at daniel.vertesy@ec.europa.eu

Further info: https://composite-indicators.jrc.ec.europa.eu/

Contributions to this presentation by Michaela Saisana and JRC colleagues in the Composite Indicators and Scoreboards (COIN) Team are gratefully acknowledged.

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission

40

Becker, W., M. Saisana, P. Paruolo, and I. Vandecasteele. 2017. ‘Weights and Importance in Composite Indicators: Closing the Gap. Ecological Indicators 80: 12–22.Cohen, A., Saisana, M., 2014, Quantifying the Qualitative: Eliciting Expert Input to Develop the Multidimensional Poverty Assessment Tool, Journal of Development Studies 50(1): 35-50.Davis, K. E., Fisher, A., Kingsbury, B. and Merry, S.E. (eds.) (2012) Governance by Indicators: Global Power Through Quantification and Rankings. Oxford: Oxford University Press.Ferretti, F., Guimaraes Pereira, A., Hardeman, S. & Vertesy, D. (2017) “Research Excellence Indicators: time to re-imagine the ‘making of’?” Paper under revision for Science and Public PolicyOECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD Publishing, ISBN 978-92-64-04345-9. Paruolo P., Saisana M., Saltelli A., 2013, Ratings and Rankings: voodoo or science?. J Royal Statistical Society A 176(3), 609-634.Power, M. (1994). The audit explosion (No. 7). Demos Paper No. 7. Rottenburg, R., Merry, S.E., Park, S-J. and Mugler, J. (eds.) (2015) The World of Indicators: The Making of Governmental Knowledge through Quantification. Cambridge: Cambridge University PressSaltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81(1), 65-77.Saisana, M., and Saltelli, A., 2011, Rankings and Ratings: Instructions for use, Hague Journal on the Rule of Law 3(2), 247-268.

References and related reading

41

Saisana M., D’Hombres B., Saltelli A., 2011, Rickety Numbers: Volatility of university rankings and policy implications. Research Policy 40, 165–177.Saisana, M., Saltelli, A., 2014, JRC Statistical Audit of the WJP Rule of Law Index® 2014 (p.188-198). In the World Justice Project Rule of Law Index® Washington, D.C.: The World Justice Project.Saisana, M., Weziak-Bialowolska, D., 2013, JRC Statistical Audit on the Environment and Gender Index (p.143-153), in IUCN The Environment and Gender Index (EGI) 2013 Pilot. Washington, D.C.: IUCN.Saisana, M., Saltelli A., 2012, Corruption Perceptions Index 2012. Statistical Assessment, EUR 25623, European Commission, JRC-IPSC, Italy.Saisana M., 2010, ELLI-Index: a sound measure for lifelong learning in the EU, EUR 24529, European Commission, JRC-IPSC, Italy.Saisana M., Saltelli A., 2010, The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and Critical assessment, EUR 24310, European Commission, JRC-IPSC, Italy.Saisana M., 2008, The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, EUR 23274, European Commission, JRC-IPSC, Italy.Saisana M., D’Hombres B., 2008, Higher Education Rankings: Robustness Issues and Critical Assessment, EUR 23487, European Commission, JRC-IPSC, Italy.Vértesy, D. 2016. ‘A Critical Assessment of Quality and Validity of Composite Indicators of Innovation’. Paper presented at the OECD Blue Sky III Forum on Science and Innovation Indicators,19–21 September 2016, Ghent, Belgium.

References and related reading