What is an emerging technology? - 2015 Science and Technology Indicators (STI) Conference

What is an emerging technology?

Science and Technology Indicators Conference – 2-4 September 2015, Lugano

Rotolo Daniele1,2, Diana Hicks2, Ben Martin1,3

1 SPRU (Science Policy Research Unit), University of Sussex2 School of Public Policy, Georgia Institute of Technology3 Centre for Science and Policy (CSAP) and Centre for Business Research,

Judge Business School, University of Cambridge

The growing interest in emerging technologies

The growing interest in emerging technologies

Source: Authors’ elaboration

Year

Num

ber o

f pub

licat

ions

● ● ● ●● ● ● ● ● ●

● ● ● ●● ●

● ●●

●

●● ●

●●

●●

●●

●● ●

1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

050

100

150

200

250

300

Year

050

010

0015

0020

0025

0030

0035

00N

umbe

r of n

ews

artic

les

●

Publications in all disciplinesPublications in social sciencesNews articles

The growing interest in emerging technologies… Why?

Emerging technologies are conceived as new technologies with the potential to change the status quo

The problem

Growing interest, but no consensus on what qualifies a technology to be ‘emergent’

• Proposed definitions overlap, but also point to different characteristicso extensive socio-economic impact (e.g. Porter et al., 2002)

o long term impact (15-year horizon or so) (Porter et al., 2002)

o uncertainty (Boon & Moors, 2008)

o novelty and growth (Small et al., 2014)

• A variety of methodological approaches has been developed, especially in scientometrics, for the detection and analysis of emergence in science and technology

• These methods however build on relatively loose definitions of emerging technologies or often no definition at all is provided – methods tend to greatly differ also with the use of the same or similar techniques

Research aim

To develop a definition of ‘emerging technologies’ and a framework for their operationalisation

Defining emerging technologies

Defining emerging technologies

To develop a definition of ‘emerging technology’ our approach builds on:

1.The examination of the concept of emergence

2.The review of major innovation studies focused on technological emergence

1. The concept of emergence

Table 1: Dictionary definition of emerge/emergent.

Dictionary definition of ”emerge”/”emergent” Attributes/features

”the process of coming into being, or of becoming important and promi-nent” (New Oxford American Dictionary)

important; prominent

”to become manifest: become known [...]” (Merriam-Webster’s Colle-giate Dictionary)

become manifest; become known

”to rise up or come forth [...] to become evident [...] to come intoexistence” (The American Heritage Desk Dictionary and Thesaurus)

evident; come into existence

”move out of something and become visible [...] come into existence orgreater prominence [...] become known [...] i the process of coming intobeing or prominence” (Concise Oxford English Dictionary)

visible; prominent; become known;come into being

”starting to exist or to become known [...] to appear by coming outof something or out from behind something (Cambridge DictionariesOnline)

become known; to appear

Source: search performed by authors on major English dictionaries.

2 The concept of emergence

The common sense notion of ”emerge” or ”emergent” refers to ”the process of coming into being,

or of becoming important and prominent” (New Oxford American Dictionary) or ”to rise up or

come forth [...] to become evident [...] to come into existence” (The American Heritage Desk

Dictionary and Thesaurus). Table 1 presents variations on the dictionary definition of emergent.

The primary attribute of emergence is ”becoming” — that is, coming into existence. Emergent

is not a static property, it is a label for a process. The endpoint of the process is variously

described as visible, evident, important or prominent. Thus, among the dictionaries there is

some disagreement as to whether acknowledged existence is enough for emergence, or beyond

that, a certain level of prominence is needed in order to merit application of the term emergence.

There is a second definition of emergent given the by The New Oxford American Dictionary

as: a property arising as an e↵ect of complex causes and not analysable simply as the sum of their

e↵ects. An additional definition is: arising and existing only as a phenomenon of independent

parts working together, and not predictable on the basis of their properties. This approach to

the concept of emergence is used in the study of complex systems. The concept of emergence

can indeed be traced back to the 19th Century in the proto-emergentism movement when Lewes

(1875) referred to ’emergent e↵ects’ in chemical reactions as those e↵ects that cannot be reduced

to the components of the system, i.e. the e↵ects for which it is not possible to trace all the steps

of the processes that produced them. Its application in the study of the dynamics of complex

systems in physics, mathematics, and computer science gave rise to other fundamental theories

5

2. Review of innovation studies

• We searched for ”emerg* technolog*”, ”tech* emergence”, ”emergence of* technolog*” or ”emerg* scien* technol*”in publication titles (SCOPUS)

• 2,201 publications were identified of which 501 in social science domains

• ~ 50% of these were not relevant (focus on specific industrial context or on the educational sector)

• A core set of 12 studies that contributed to the conceptualisation of technological emergence was identified

Table 3: Definitions of emerging technologies (studies are chronologically ordered).

Study Domain Definition (elaborated or adopted)

Martin (1995) S&T policy ”A ’generic emerging technology’ is defined [...] as a technology theexploitation of which will yield benefits for a wide range of sectors ofthe economy and/or society” (p. 165)

Day andSchoemaker

(2000)

Management ”emerging technologies as science-based innovation that have the po-tential to create a new industry or transform an existing ones. Theyinclude discontinuous innovations derived from radical innovations [...]as well as more evolutionary technologies formed by the convergence ofpreviously separate research streams” (p. 30)

Porter et al.(2002)

S&T policy ”Emerging technologies are defined here as those that could exert muchenhanced economic influence in the coming (roughly) 15-year horizon.”(p. 189)

Corrocheret al. (2003)

Evolutionaryeconomics

”The emergence of a new technology is conceptualised [...] as an evo-lutionary process of technical, institutional and social change, whichoccurs simultaneously at three levels: the level of individual firms orresearch laboratories, the level of social and institutional context, andthe level of the nature and evolution of knowledge and the related tech-nological regime.” (p. 4)

Hung andChu (2006)

S&T policy ”Emerging technologies are the core technologies, which have not yetdemonstrated potential for changing the basis of competition” (p. 104)

Boon andMoors (2008)

S&T policy ”Emerging technologies are technologies in an early phase of develop-ment. This implies that several aspects, such as the characteristics ofthe technology and its context of use or the configuration of the actornetwork and their related roles are still uncertain and non-specific” (p.1915)

Srinivasan(2008)

Management ”I conceptualize emerging technologies in terms of three broad sub-heads: their sources (where do emerging technologies come from?),their characteristics (what defines emerging technologies?) and theire↵ects (what are the e↵ects of emerging technologies on firms’ strate-gies and outcomes?).” (p. 634)

Cozzens et al.(2010)

S&T policy ”Emerging technology — a technology that shows high potential buthasn’t demonstrated its value or settled down into any kind of consen-sus.” (p. 364) ”The concepts reflected in the definitions of emergingtechnologies, however, can be summarised four-fold as follows: (1) fastrecent growth; (2) in the process of transition and/or change; (3) mar-ket or economic potential that is not exploited fully yet; (4) increasinglyscience-based.” (p. 366)

Stahl (2011) S&T policy ”[...] emerging technologies are defined as those technologies that havethe potential to gain social relevance within the next 10 to 15 years.This means that they are currently at an early stage of their develop-ment process. At the same time, they have already moved beyond thepurely conceptual stage. [...] Despite this, these emerging technologiesare not yet clearly defined. Their exact forms, capabilities, constraints,and uses are still in flux” (p. 3-4)

Alexanderet al. (2012)

S&T policy ”Technical emergence is the phase during which a concept or constructis adopted and iterated by [...] members of an expert community ofpractice, resulting in a fundamental change in (or significant extensionof) human understanding or capability.” (p. 1289)

Halaweh(2013)

Management Characteristics of (IT) emerging technologies ”are uncertainty, networke↵ect, unseen social and ethical concerns, cost, limitation to particularcountries, and a lack of investigation and research.” (p. 108)

Small et al.(2014)

Scientometrics ”[...] there is nearly universal agreement on two properties associatedwith emergence — novelty (or newness) and growth.” (p. 2)

Source: search performed by authors on SCOPUS and extended to cited references.

9

Attributes of emergence

2. Review of innovation studies

We analysed the textual content of the proposed definitions to extract all the component conceptsand grouped those into attributes of emergence

Table 4: Attributes of emergence and reviewed key innovation studies.

Innovation studies defining emerging technologies

Attribute of emergence Martin(199

5)

Day

andSch

oem

aker

(200

0)

Porter

etal.(2002)

Corroch

eret

al.(200

3)

Hungan

dChu(200

6)

Boon

andMoors(200

8)

Srinivasan

(200

8)

Coz

zenset

al.(201

0)

Stahl(201

1)

Alexan

der

etal.(201

2)

Halaw

eh(201

3)

Smallet

al.(201

4)

Radical novelty x x x

Relatively fast growth x x x

Coherence x x x x

Prominent impact x x x x x x x x x

Uncertainty and ambiguity x x x x x x x

Source: authors’ elaboration.

domain in which it is arising. However, the impact the technology can exert on that domain is

still relatively low. The technology has not yet gone beyond the purely conceptual stage. It is

still not a coherent whole. It is likely that multiple designs exist and multiple communities are

involved in its development — the delineation of the boundary of the technology is particularly

problematic at this stage. As a consequence, its growth is relatively slow or not yet begun,

and high levels of uncertainty and ambiguity are associated with the future developments of the

technology — the technology may not even emerge. In the emergence phase, the technology

acquires a certain momentum. Attributes of emergence are subject to dramatic change. The

technology becomes more coherent. Some trajectories of development may have been selected

out and certain dimensions of performance prioritised and improved. A community of practice

may have also emerged. The impact the technology may exert is rather less uncertain and

ambiguous, and the technology starts to take o↵ in terms of publications, patents, researchers,

firms, prototypes/products, etc. However, at the same time, it is likely that the radical novelty

of the technology will diminish — other technologies that exploit di↵erent basic principles may

be emerging as well in the domain in which the considered technology is emerging. Finally,

in the post-emergence phase, impact and growth may enter a declining phase, the technology

loses its radical novelty, and knowledge of the possible outcomes of the technology becomes

more complete (probabilities can be perhaps assigned to outcomes). In line with the S-shaped

patterns highlighted in early studies on the growth of science (e.g De Solla Price, 1963) and in

14

Radical novelty

• Included in 2/12 definitions: ”novelty (or newness)” (Small et al., 2014),”discontinuous innovations derived from radical innovations” (Day & Schoemaker, 2000)

• To achieve a new or a changed purpose/function, emerging technologies build on different basic principles (e.g. cars with an internal combustion engine vs. an electric engine) (Arthur, 2007)

• Revolutionary/evolutionary technologies and radical novelty (see Adner & Levinthal, 2002)

o ‘Revolutionary’ – technologies with relatively limited prior developments (nano-materials, DNA sequencing)

o ‘Evolutionary’ – niches and speciation process (e.g. wireless communication technology), but also incremental technological advances

• ‘Novelty’ vs. ‘radical novelty’

Relatively fast growth

• Included in 3/12 definitions: ”fast clock speed” (Srinivasan,

2008) or ”fast growth” (Cozzens et

al., 2010), or ”growth” (Small et al., 2014)

• Growth may be observed across a number of dimensions (e.g. actors, public and private funding, publications, patents, prototypes, products)

• The context matters: A technology may grow rapidly in comparison with other technologies in the same domain(s) which may be growing at a slower pace Source: Consumer Electronic Association (2011)

Coherence

• Included in 4/12 definitions: ”convergence of previously separated research streams” (Day &

Schoemaker, 2000), ”convergence in technologies” (Srinivasan, 2008), technologies that ”have already moved beyond the purely conceptual stage” (Stahl, 2011)

• Also as arising of ”an expert community of practice” that adopts and iterates the concepts or constructs” (Alexander et al., 2012)

o Both a number of people and a professional connection between those people are necessary – coherence refers to internal characteristics of a group such as ’sticking together’, ’being united’, ’logical inter- connection’ and ’congruity’

o The status of external relations is also important – the emerging technology must detach itself from its technological ’parents’ to some degree to merit a separate identity (Glanzel and Thijs, 2012)

Prominent impact

• Included in 9/12 definitions: ”benefits for a wide range of sectors” (Martin, 1995),”create new industry or transform existing ones” (Day & Schoemaker, 2000),or change ”the basis of competition” (Hung & Chu, 2006), etc.

• This conceptualisation inevitably excludes technologies that may still exert a prominent impact within specific domains

• ‘Scope’ of a technology: few vs. many domains (‘GPT’) of applications

Uncertainty and ambiguity

• ‘Uncertainty’ (identified in 6/12 definitions) is generally expressed in terms of the ’potential’ that emerging technologies have for changing the existing ’ways of doing things’ (e.g. Boon & Moors, 2008; Cozzens et al., 2010)

EMBO reports VOL 8 | NO 4 | 2007 ©2007 EUROPEAN MOLECULAR BIOLOGY ORGANIZATION310

science & society ta lking point

by applying the same battery of techniques: quantifying and aggregating different out-comes and multiplying by their respective probabilities to yield a single reductive picture of ‘risk’.

Fig 1 also provides examples of areas in which these possible states of knowledge might come to the fore in policy-making. In the top left of the matrix exist many fields in which past experience or scientific models

are judged to foster high confidence in both the possible outcomes and their respective probabilities. In the strict sense of the term, this is the formal condition of risk and it is under these conditions that the conven-tional techniques of risk assessment offer a scientifically rigorous approach. However, it is also clear that this formal definition of risk also implies circumstances of uncer-tainty, ambiguity and ignorance under which the reductive techniques of risk assessment are not applicable.

Under the condition of uncertainty (Fig 1), we can characterize possible out-comes, but the available information or analytical models do not present a definitive basis for assigning probabilities. It is under these conditions that “probability does not exist” (de Finetti, 1974). Of course, we can still exercise subjective judgements and treat these as a basis for systematic analy-sis. However, the challenge of uncertainty is that such judgements might take several different—yet equally plausible—forms. Rather than reducing these to a single value or recommendation, the scientifically rigor-ous approach is therefore to acknowledge various possible interpretations. The point remains that, under uncertainty, attempts to assert a single aggregated picture of risk are neither rational nor ‘science-based’.

Under the condition of ambiguity, it is not the probabilities but the possible out-comes themselves that are problematic. This might be the case even for events that are certain or have occurred already (Wynne, 2002; Stirling, 2003). For exam-ple, in the regulation of genetically modi-fied (GM) food, such ambiguities arise over ecological, agronomic, safety, economic or social criteria of harm (Stirling & Mayer, 1999). When faced with such questions over “contradictory certainties” (Thompson & Warburton, 1985), rational choice theory has shown that analysis alone is unable to guarantee definitive answers (Arrow, 1963). Where there is ambiguity, reduction to a single ‘sound scientific’ picture of risk is also neither rigorous nor rational.

Finally, there is the condition of igno-rance. Here, neither probabilities nor outcomes can be fully characterized (Collingridge, 1980). Ignorance differs from uncertainty, which focuses on agreed, known parameters such as carcinogenicity or flood damage. It also differs from ambi-guity in that the parameters are not only contestable but also—at least in part—unknown. Some of the most important

Fig 1 | Contrasting states of incomplete knowledge, with schematic examples. TSE, transmissible spongiform encephalopathy.

Coal

Oil

Gas

Nuclear

Hydro

Wind

Solar

Biomass

Key

RISK – as economic ‘externality’ (cUS/kWh)

Lowestresult

Medianresult

75%-ile25%-ile Highestresult

0.001 0.1 10 1000

n= 36

n= 20

n= 31

n= 21

n=16

n=18

n=11

n= 22

Fig 2 | Practical limits to robustness in risk assessment. Results were obtained from 63 detailed risk–benefit and cost–benefit comparative studies of electricity supply. Based on data from Sundqvist et al, 2004.

■ RISK

■ Familiar systems

■ Controlled conditions

■ Engineering failure

■ Known epidemics

■ Transport safety

■ Flood (under normal conditions)

■ UNCERTAINTY

■ Complex, nonlinear, open systems

■ Human element in causal models

■ Specific effects beyond boundaries

■ Flood under climate change

■ Unassessed carcinogens

■ New variant human pathogens

IGNORANCE ■ Unanticipated effects ■ Unexpected conditions ■ Gaps, surprises, unknowns ■ Novel agents like TSEs ■ Novel mechanisms ■ such as endocrine disruption

AMBIGUITY ■ Contested framings, questions, ■ assumptions, methods

Comparing incommensurables: ■ apples and oranges

Disagreements between ■ specialists, disciplines

Issues of behaviour, ■ trust and compliance

Interest, language, meaning ■ Matters of ethics and equity ■

NOT problematicNOT problematic

Problematic

Problematic

Knowledge aboutPROBABILITIES

Knowledge aboutOUTCOMES

Source: Stirling (2007)

… so, what is an emerging technology?

An emerging technology is a radically novel and relatively fast growing technology characterised by a certain degree of coherence persisting over time and with the

potential to exert a considerable impact on the socio-economic domain(s) which is

observed in terms of the composition of actors, institutions and patterns of

interactions among those, along with the associated knowledge production

processes.

Its most prominent impact, however, lies in the future and so in the emergence

phase is still somewhat uncertain and ambiguous.

A framework for the

operationalisation of emergence

Contemporary analysis Retrospective analysis

Radical novelty

Content analysis of• news articles• editorials• reviews

Citation and co-word analyses• New clusters linking otherwise weakly connected

clusters (e.g. Furukawa et al., 2015) or citing more recent clusters (e.g. Morris et al., 2003)

• Clusters that are new to both the co-citationand the direct citation model (Small et al., 2014)

Overlay mapping• Use of new knowledge bases (Rafols et al., 2010)


Not yet observed in scientometric data Indicators and trend analysis• Yearly count of documents (including modelling)• ’Bursts of activity’ (Kleinberg, 2002)• Increasing number of authors (Bettencourt et al., 2008)

Citation and co-word analyses• Growth of clusters (e.g. Ohniwa et al., 2010)

Coherence Indicators and trend analysis• Appearance of abbreviations (Reardon,

2014) and categories (Cozzens et al., 2010)• Creation of conference tracks, journal SI,

and new journals (Leydesdorff et al., 1994)

Indicators and trend analysis• Entropy measures (e.g. Watts & Porter, 2003)• Dense sub-graphs in co-authorship networks (e.g.

Bettencourt et al., 2009)

Citation and co-word analyses• Dense sub-graphs citation/co-word networks (e.g. Yoon et

al, 2010, Furukawa et al., 2015)

Prominent impact

Not yet observed in scientometric data Impact seems to be taken for granted

Uncertainty & ambiguity

Measurement challenges Efforts in measuring uncertainty reduction with Triple-Helix models (Lucio-Arias & Leydesdorff, 2009), but this area remainslargely unexplored


Radical novelty
















Prominent impact





Radical novelty
















Prominent impact




Indicators of early emergence (e.g. altmetrics)

STS mixed qualitative-quantitative approaches to map expectations(e.g. Borup et al. 2006; van Lente &

Bakker, 2010)

Numerous scientometric approaches to generate intelligence

Novel data sources (e.g. funding data)

Discussion

Discussion

• Scientometric contribution to operationalise the attributes of emergence:o Techniques are intrinsically more effective for retrospective analyses

o Focus on the detection of what is emerging, rather than on characterising the potential of what is detected to be emerging (e.g. uncertainty and ambiguity)

o Emergence as an artifact of the used method (models, data, clustering)

• Complementarity between STS and scientometrics traditions:

o STS tradition attempts to address questions of how emergence happens, thus it may

favouring meaningful interpretations of scientometric data

o Scientometrics brings a more robust empirical approach (e.g. statistical inference)

Discussion – Future research

• Conceptualisation

o Origins of emerging technologies?

Some technologies acquire a certain momentum and enter the emergent phase, others

do not emerge at all

o When does the emergence phase stop?

Limited knowledge of the end point of the emergence process

• Operationalisation

o Counterfactual sample?

Studies often tend to analyse emerging technologies, without comparing them with a

counterfactual sample of technologies that eventually did not emerge

o Novel data sources

! Publication-full-text (e.g. sentiment analysis)

! Funding data ('uncertainty and ambiguity; relatively fast growth)

! Big data and altmetrics as ’real-time’ data for early detection indicators

Conclusions

• Considerable disagreement exists on what is technological emergence and how it should be

operationalised

• Implications for policy-making in the context of emerging technologies (e.g. resource

allocation, creation of research programmes, drawing up of regulations)

• Notwithstanding the fuzziness of the concept of emerging technologies, this study is one of

the first attempts to increase the conceptual clarity on the phenomenon – a necessary

precondition for a coherent and systematic operationalisation of emerging technologies

….thank you

Funded by:

[email protected] @danielerotolo

www.danielerotolo.com

http://dx.doi.org/10.1016/j.respol.2015.06.006

Research Policy 44 (2015) 1827–1843

Contents lists available at ScienceDirect

Research Policy

jo ur nal ho me page: www.elsev ier .com/ locate / respol

What is an emerging technology?

Daniele Rotoloa,b,∗, Diana Hicksb, Ben R. Martina,c

a SPRU – Science Policy Research Unit, University of Sussex, Brighton BN1 9SL, United Kingdomb School of Public Policy, Georgia Institute of Technology, Atlanta 30332-0345, United Statesc Centre for Science and Policy (CSAP) and Centre for Business Research, Judge Business School, University of Cambridge, Cambridge CB2 1QA,United Kingdom

a r t i c l e i n f o

Article history:Received 11 December 2014Received in revised form 15 June 2015Accepted 16 June 2015

Keywords:Emerging technologiesConceptualisationDefinitionAttributes of emergenceOperationalisationDetection and analysisFrameworkScientometricsIndicatorsScience and Technology Studies (STS)

a b s t r a c t

There is considerable and growing interest in the emergence of novel technologies, especially from thepolicy-making perspective. Yet, as an area of study, emerging technologies lack key foundational ele-ments, namely a consensus on what classifies a technology as ‘emergent’ and strong research designsthat operationalise central theoretical concepts. The present paper aims to fill this gap by developing adefinition of ‘emerging technologies’ and linking this conceptual effort with the development of a frame-work for the operationalisation of technological emergence. The definition is developed by combininga basic understanding of the term and in particular the concept of ‘emergence’ with a review of keyinnovation studies dealing with definitional issues of technological emergence. The resulting definitionidentifies five attributes that feature in the emergence of novel technologies. These are: (i) radical novelty,(ii) relatively fast growth, (iii) coherence, (iv) prominent impact, and (v) uncertainty and ambiguity. Theframework for operationalising emerging technologies is then elaborated on the basis of the proposedattributes. To do so, we identify and review major empirical approaches (mainly in, although not limitedto, the scientometric domain) for the detection and study of emerging technologies (these include indica-tors and trend analysis, citation analysis, co-word analysis, overlay mapping, and combinations thereof)and elaborate on how these can be used to operationalise the different attributes of emergence.

© 2015 Elsevier B.V. All rights reserved.

1. Introduction

Emerging technologies have been the subject of much debatein academic research and a central topic in policy discussions andinitiatives. Evidence of the increasing attention being paid to thephenomenon of emerging technologies can be found in the grow-ing number of publications dealing with the topic and news articlesmentioning emerging technologies (in their headlines or lead para-graphs), as depicted in Fig. 1. Increasing policy interest in emergingtechnologies, however, must be set against a literature where noconsensus has emerged as to what qualifies a technology to beemergent. Definitions proposed by a number of studies overlap,but also point to different characteristics. For example, certain def-initions emphasise the potential impact emerging technologies arecapable of exerting on the economy and society (e.g. Porter et al.,2002), especially when they are of a more ‘generic’ nature (Martin,

∗ Corresponding author at: SPRU – Science Policy Research Unit, University ofSussex, Brighton BN1 9SL, United Kingdom. Tel.: +44 1273 872980.

E-mail addresses: [email protected] (D. Rotolo),[email protected] (D. Hicks), [email protected] (B.R. Martin).

1995), while others give great importance to the uncertainty asso-ciated with the emergence process (e.g. Boon and Moors, 2008) orto the characteristics of novelty and growth (e.g. Small et al., 2014).The understanding of emerging technologies also depends on theanalyst’s perspective. An analyst may consider a technology emer-gent because of its novelty and expected socio-economic impact,while others may see the same technology as a natural extensionof an existing technology. Also, emerging technologies are oftengrouped together under ‘general labels’ (e.g. nanotechnology, syn-thetic biology), when they might be better treated separately giventheir different socio-technical features (e.g. technical difficulties,involved actors, applications, uncertainties).

The lack of consensus over definitions is matched by an ‘eclec-tic’ and ad hoc approach to measurement. A wide variety ofmethodological approaches have been developed, especially by thescientometric community, for the detection and analysis of emer-gence in science and technology domains (e.g. Porter and Detampel,1995; Boyack et al., 2014; Glänzel and Thijs, 2012). These methods,favoured, because they take advantage of growing computationalpower and large new datasets and allow one to work with moresophisticated indicators and models, lack strong connections towell thought out concepts that one is attempting to measure, a basic

http://dx.doi.org/10.1016/j.respol.2015.06.0060048-7333/© 2015 Elsevier B.V. All rights reserved.