Different views on scholarly talent

What are the talents we are looking

for in science

Pleun van Arensbergen1 Inge van der Weijden2 andPeter van den Besselaar3

1Nijmegen School of Management Radboud University Nijmegen The Netherlands 2Centre for

Science and Technology Studies CWTS Leiden University Leiden The Netherlands and 3Network

Institute amp Department of Organization Science VU University Amsterdam The Netherlands

Corresponding author Email pvanarensbergenfmrunl

In this article we study the evaluation of talented early career researchers as done in grantallocation processes To better understand funding decisions we studied the grant allocationprocess in more detail and compared the notion of talent in grant allocation with moregeneral notions of talent existing in the academic work environment The comparison is basedon interviews with 29 scholars who have experience with identifying talent both in their dailyacademic work and in the process of grant allocation Overall there is large agreement on thenotion of talent However the characteristics ascribed to top talent vary depending on the evalu-ation context In grant allocation a narrower talent definition prevails compared with more generalevaluation Furthermore difficulties arise in the process of panel decision-making when selectioncriteria need to be concrete and explicit to enable comparison Having to choose between manyapplicants of similar quality makes the selection process liable to subjectivity arbitrariness andrandomness Despite these uncertainties grants are ascribed a very high symbolic value Smallquality differences are enlarged into considerable differences in recognition consequently affect-ing career opportunities as they provide academics with both financial and symbolic resources

Keywords talent selection grant allocation peer review symbolic capital academic careers

1 Introduction

The quality of higher education and research is stronglyconnected to the quality of the people working in theacademic sector For excellent science excellent scientistsare needed Therefore government and universities specif-ically aim at selecting the best people when investing publicresources in education and research This has led to anincreased focus on talent and talent policy especiallywithin the group of early career researchers This is ageneral phenomenon also clearly visible within theNetherlands the specific context of this study In thehiring policies of Dutch universities the notion of talentnowadays has taken a central position It is becoming akey issue in Higher Education for Human ResourceManagement a management tool that focuses on individ-ual performance (Waring 2013) Several programs and

policy initiatives are currently implemented to attractand stimulate academic talent eg the Tenure Trackprogram scholarships for excellent PhD students andmentoring programs for promising female academics(De Boer and Jongbloed 2010 Van den Brink Fruytierand Thunnissen 2012) Furthermore many programs aredirected towards motivating researchers to apply forexternal grants and to increase the chances of young re-searchers to acquire external funding They involve prese-lection processes encouragement by dedicated mentorstraining and supervision of writing grant applicationsand improvement of presentation skills (Neufeld Huberand Wegner 2013)

The increased focus on acquiring external funding isboth a deliberate consequence of changes in fundingpolicy (from institutional towards project funding) as itis a result of the growing pool of competent scholars with

Research Evaluation (2014) pp 1ndash12 doi101093resevalrvu015

The Author 2014 Published by Oxford University Press All rights reserved For Permissions please email journalspermissionsoupcom

Research Evaluation Advance Access published August 20 2014

academic career ambitions Government funding used tobe predominantly allocated as block funding to the entireuniversity or research institute Nowadays in manycountries intermediaries like research councils have takenan important position in the distribution of resources asthey allocate an increasing share of government fundingdirectly to individual researchers or research groups(Lepori et al 2007) Securing external research grantshas become a prominent criterion in academic recruitmentevaluation and promotion processes in science (De JongeAkademie 2010 Laudel and Glaser 2012 VanArensbergen Hessels and Van der Meulen 2013 BlochGraversen and Pedersen 2014) Therefore careeropportunities as well as university appointment decisionsof individual scientists depend on granting decisions madeby external funders like the European Research Council(ERC) or the Dutch Research Council (NWO) Personalcareer grants are to an increasing extent considered as anecessary resource to further develop an academic careerThe real effect of these grants on someonersquos career issubject of many studies A recent study (Laudel andGlaser 2012) on the impact of ERC grants on careers ofgrantees showed that several organizations responded tothe reputation of ERC grants by promoting (mainly ERCstarting) grantees or by offering them tenure At the sametime the grants were found to only play a minor role inpromoting interorganizational mobility Most of thegrantees already worked in the best possible environmentsandor were settled with their families which constrainedthem to move A Danish study showed that grant recipi-ents from the Danish council for Independent Researchhave a higher probability of becoming a full professor(16) compared with rejected applicants (9) Alsogrant recipients stressed the central role of grants infacilitating subsequent collaboration with leading re-searchers in their field and in establishing their own pos-itions in research communities (Bloch et al 2014)

A successful personal funding instrument of the DutchResearch Council is the Innovational Research IncentivesScheme (Vernieuwingsimpuls) Gerritsen Plug and Van derWiel (2013) recently studied the effect of these grants andfound that they indeed increase onersquos chance of a success-ful academic career Comparing applicants with about thesame priority scores they found that grantees are morelikely than rejected applicants to stay in academiareceive follow-up grants and become a full professorThese results show that the allocation of personal careergrants is an important context of talent selection To studythe notion of talent within science the process of grantallocation is therefore included and investigated in moredetail

11 Research questions

In this study we investigate the process of talent selectionin more detail with a twofold aim On the one hand we

want to create a better understanding of the notion ofacademic talent What are the talents the academic com-munity is looking for both within academia in general aswithin grant allocation procedures On the other hand wewant to open up the black box of grant allocation by sci-entific panels For improving the transparency qualityand legitimacy of grant allocation practices it would there-fore be important to uncover the details of the de facto(implicit and explicit) applied criteria Which characteris-tics of applicants do the panel reviewers value the most andhow do they reach agreement within panels

12 Theoretical background

The word lsquotalentrsquo clearly has a positive connotation butthere is no general consensus on the exact meaning of it Ahighly debated issue for example is the origin of talent istalent innate or acquired (eg Baron-Cohen 1998 vsHowe Davidson and Sloboda 1998) A recent quantitativestudy on career grant allocation showed that except for afew positive outliers (top talents) no evident pool oftalents could be identified based on review scores (Vanden Besselaar and Van Arensbergen 2013) Thomas andNedeva (2013) recently developed a multidimensionalframework of 23 elements to characterize talented re-searchers based on an extensive literature reviewExamples of these elements are geographic and workplacemobility demographic variables and the amount ofacademic service tasks undertaken

121 Symbolic capital In this article we relate thenotion of talent to lsquosymbolic capitalrsquo (Bourdieu 1986)Bourdieu discerned several forms of capital most import-antly economic (financial resources) cultural (educationand upbringing) and social (relations and networks)capital In every field like politics arts economy andscience there is competition for accumulating as muchcapital as possible The general distribution of thesetypes of capital and the way they can be accumulated isfield specific The various forms of capital can be convertedinto other forms most importantly into symbolic capitalSymbolic capital concerns the reputation and prestige ac-credited to someone based on the recognition of hisaccumulated capital In science this is mainly determinedby the judgment of peers Evaluation processesmdashwhetherit concerns reviewing scientific manuscripts for publicationor assessing academics for recruitment or promotion prac-ticesmdashcan be characterized as self-governing as they aremainly executed by academics themselves Academic repu-tation and quality therefore lives by peer recognition Werelate symbolic capital to talent selection because assigningthe prestigious label of talent can be seen as recognition ofa personrsquos accumulated capital

What is considered to be valuable capital and how itcan be accumulated are determined by the academic

2 of 12 P van Arensbergen et al

habitus According to Bourdieursquos concept of habitus aca-

demics generally have internalized certain dispositions and

norms in response to objective conditions they encoun-

tered through their academic work The habitus is

shaped by past experiences and guides current behaviour

and thinking within a specific field It refers to the embodi-

ment of certain dispositions so not only at the explicitconscious level (Bourdieu 1988) From the perspective of

talent selection habitus determines what is considered to

be valuable capital as it relates to dispositions academics

should have or should develop in time People are not only

assessed on their current dispositions but also on their po-

tential what are they expected to become reflecting the

habitus of the more established academics Looking into

how academics decide who is talented and who is not will

provide more insight in which qualities of academics the

scientific community value the most and are part of the

academic habitusIn this study we modify Bourdieursquos forms of capital and

differentiate between professional individual and social

capital following the example of Van den Brink (2009)

who applied the concept of symbolic capital to appoint-

ment practices within academia Professional capital

involves skills experience and achievements related toresearch teaching and management This is mainly

assessed from track records eg in terms of years of ex-

perience former employers number of publications and

acquired funding Formal assessment criteria predomin-

antly reflect professional capital Individual capital has

more of a subjective character and is about personal

traits and motivation For example creativity persever-

ance commitment and likeability relate to individual

capital Finally social capital is defined as consisting lsquoof

an aggregation of networks and these networks provide

access to certain resources and positions of powerrsquo (Van

den Brink 2009 145) Social capital is not only valuable

because it itself can be converted into symbolic capital but

also because it serves as an accelerator for turning

accumulated professional and individual capital into

symbolic capital academic prestige As science is turning

more into a social activity and as collaboration is of

growing importance we chose to extend the concept ofsocial capital with those skills and traits needed for inter-

action and collaborationmdashthe skills needed for creating

and maintaining social networks Thus some of the skills

that generally are considered as professional or individual

capital (eg communication skills ability to collaborate

and social attitudes) are treated in this study as (conditions

for) social capitalTo summarize in this article we study symbolic cap-

ital in the context of talent selection as the recognition

of professional individual and social capital We will

identify the skills and traits characterizing talent and

categorize them as one of the three types of capital (see

Table 3)

122 Academic talent selection An inventory of defin-itions of talent in policy documents of several Dutchuniversities (Thunnissen Fruytier and Van den Brink2010) shows that the general descriptions of talent leaveconsiderable room for interpretation talents are peoplewho perform better than expected based on their age andor experience (p 20) or talents are they of whom is expectedto be able to shortly acquire a position as an associate pro-fessor (p 19) The main criteria for assessing young talentsare publications study and promotion results honoursdegrees awards grants and international experienceThese criteria primarily relate to professional capital Toa lesser extent individual capital is mentioned withinpolicy documents such as motivation and drive

Evaluation of scientific quality is often carried out inpanels To understand how talent is evaluated andselected in these panels it is not enough to only studycharacteristics of the talents and the reviewers Panel deci-sions are influenced by and the result of group inter-action making group and context characteristicsimportant variables to include Luukkonen (2012)studied review processes of ERC panels and showed howpanel decisions are steered by customary interpretative anddeliberation rules From literature reviews on this type ofpanel reviewing we learn that for example group com-position group dynamics (eg discussion sharing of infor-mation power relations) characteristics of the procedureand contextual factors (eg budget time pressureaccountability) can strongly affect the decision outcomes(Olbrecht and Bornmann 2010 Van Arensbergen Van derWeijden and Van den Besselaar forthcoming) Thesefactors impede the transparency and predictability ofdecision-making processes However as this type ofpanel evaluation involves interactions between humanbeings it needs to be considered as a social and emotionalprocess Therefore it is impossible to completely rule outany form of subjectivity (Lamont 2009)

In this article we approach the process of talent selec-tion as a strongly subjective process We study it as fully aspossible by investigating the decision-making process onboth the individual and group level We look at how re-viewers use formal procedures and interpret formal criteriain their own way when evaluating grant applicationsFurthermore we analyse panel discussions and the waypanels reach their final allocation decisions By studyingthe process of personal grant allocation we aim to get abetter understanding of how symbolic capital is ascribedwithin academic talent selection

2 Data and methods

Data for this article consist out of 29 semi-structured inter-views with members of grant panels All the intervieweeswere involved in reviewing and allocating personal careergrants in 2009 for two funding programs within the Talent

Different views on scholarly talent 3 of 12

Program called Innovational Research Incentives Scheme(Vernieuwingsimpuls) of the NWO Table 1 gives anoverview of these funding programs the Early CareerGrant (ECG) and Intermediate Career Grant (ICG)scheme

This grant scheme is not limited to a certain scientificdomain but has eight domain panels and one interdiscip-linary panel In this article we grouped the eight domainsinto two main domains Social Sciences and Humanities(SSH) and Natural and Technical Sciences (NTS)

From the list of about 220 panellists active in 2009 weinvited 40 of them across the various domains for an inter-view Efforts were made to attain comparable numbers ofwomen and men The large majority was willing to partici-pate five people did not reply and seven were willing butunable to participate because of time constraints or illnessThe interviewees are predominantly associates or full pro-fessors and come from various scientific domains fromsocial sciences to life sciences (see Table 2 for moredetails) Most of the interviewees have been involved inthis type of grant allocation for several years have experi-ence in internal selection processes at the university andhave been active as (national and international) peerreviewers

The research fieldwork was conducted in the beginningof 2012 Most of the interviews were conducted at theintervieweersquos workplace (27) two were done by telephonewith an average duration of 1 hour Interviewees wereasked about identification of talent in their dailyacademic work (eg describe young academics thatclearly stood out in their group or who they really wouldhave liked to retain) As this type of identification concernsthe intervieweersquos general view on talent apart from anyspecific selection or appraisal procedures we refer to thisas general talent identification We feel this provides auseful frame of reference to better understand specifictypes of talent selection Then we asked about talent selec-tion within the grant panel (eg how do they review thegrant applications what criteria do they use in the differ-ent phases of the selection process how do they recognizethe top talents how are the applicants discussed within thepanel how does the panel reach the final allocation deci-sions) This we will call concrete grant talent selectionbecause of the presence of concrete selection proceduresand because actual selection takes place

All interviews were audiotape recorded transcribed ver-batim sent back to interviewees for authorization andcoded using the software program Atlasti (see Table 3for a summary of this code scheme)

3 Results

First we focus on how talent is generally identified in dailywork at the university before shifting to talent selection inthe concrete context of grant allocation At the end of this

article we will look into the process of group decision-making and which difficulties the panellists face in thefinal phase of the selection process As our main focus isnot on gender career stage or disciplinary differences wewill not structurally compare men and women nor theECG and ICG schemes nor the NTS and SSH domainswhen describing our results but only where we foundimportant differences

31 The general concept of scholarly talent

Real talents were said to be easily and widely recognizedThey are those who do not excel on one single dimensionbut on various dimensions combining all forms of capitalTalent is multidimensional Related to professionalcapital talented academics distinguish themselves fromothers by having broad expertise excellent writing skillsand above all a high productivity Productivity is posi-tively assessed when one has a high number of publica-tions also first authored preferably in prestigiousjournals with a high impact factor This is in line withthe formal criteria formulated in many career policydocuments

Although young academics need to have accumulatedconsiderable professional capital to be identified as

Table 2 Overview of the interviewees per program domain and

gender

ECG ICG

Male Female Male Female Total

Social Sciences and Humanities 4 5 2 3 14

Natural and Technical sciences 7 3 2 2 14

Total 11 8 5a 5 29a

aIncluding one panellist for cross disciplinary applications which is not categorized

as SSH or NTS

Table 1 Innovational research incentives scheme

ECG (Veni) ICG (Vidi)

Career conditions 0ndash3 years after PhD 0ndash8 years after PhD

Funding Max E250000

per grant

Max E800000

per grant

Duration 3 years 5 years

Number of grants 150 per year 85 per year

Acceptance

rates ()

2002 25 17

2005 22 26

2005 18 21

2012 16 14

2013 15 20

Source wwwnwonlvi

4 of 12 P van Arensbergen et al

talent intervieweesrsquo emphasized talents above all stand outfrom the majority because of their individual capitalEspecially female interviewees strongly value this type ofcapital as more than half of all traits mentioned by womenhad to do with personality and motivation The personal-ity trait that is central to the academic habitus and men-tioned most frequentlymdashby both men and womenmdashis theability to work very hard People who lack this ability willnever make it in science especially in top science This alsomeans young researchers need to be very ambitious andeager possess a strong drive to become a top researcherand work with great enthusiasm and passion Because theacademic world is a competitive world and setbacks areunavoidable perseverance is considered to be an exigencya crucial part of the academic habitus Top talents do notlet themselves be demotivated by rejections and negativereviews but they learn from them and use these experi-ences to grow Furthermore talents are willing to learnand are strongly self-conscious and goal directed A lastpersonal trait which seems to be more important withinNTS compared with SSH is the ability to work independ-ently and lsquothink for yourselfrsquo

Being a social person is found to be a primary require-ment to be considered a talent As science is not purely anindividual activity but becomes to an increasing extent asocial activity it is found of great importance to fit well ina group and to have good social skills Social skills seem tobecome a more important part of the academic habitusover time In line with Bourdieursquos meaning of socialcapital young talents are described as people whoalready created their own significant network and are notjust embedded in their professorrsquos network1 Having astrong network is seen as meaningful in two ways Itshows they are good academics as they are acknowledgedby other people who they are connected with It is alsoconsidered a strategy to increase your career opportunitiesand academic success If you really want to be successful inscience you need to have the right connections and theability to make and maintain these connectionsInterestingly having your own network as characteristic

of talent was only mentioned by women not by men

This could be explained by earlier research (Bleijenbergh

Benschop and Vennix 2013 Van den Brink 2009) suggest-

ing that networks play an important role in distribution of

information resources and career opportunities As men

are over-represented in academia and have more homo-

genous networks than women (Brass et al 2004) women

may be disadvantaged and therefore more aware of the

importance of having a good networkOther social characteristics valued in talents are having a

strong team spirit being interested in others willing to

help others and being able to motivate them Finally

talents have good communication skills take a lot of ini-

tiatives and they are proactive

On the one hand you want to hire the best people but they shouldalso fit in the group Some people are very good but completelyantisocial [ ] we are all just human beings and we want

someone in our group who is a nice person There are plenty ofpeople with whom in theory I could collaborate with perfectlycontent wise But the chemistry is not right or there are little

things of which you think am I going to invest my time in this(male interdisciplinary)

To sum up the majority of characteristics of talent

valued in general daily academic work can be described

as features of individual capital personal traits motiv-

ation and ambition For women this holds even stronger

than for menHowever one type of professional capital was found to

be of special importance acquired grants A large majority

of the interviewees do not consider these simply as part of

track records but as a significant indicator in itself of

previous recognition of talent In times of severe competi-

tion and generally low allocation rates grants have an

important value They provide the receiver with both

financial resources and prestige Especially personal

career grants as the ERC grants and the Dutch Veni-

and Vidi-grants are considered very meaningful for

young researchers

Table 3 Most important codes describing the three types of capital

Professional Individual Social

Awards

Broad expertise

Clear presentation (interview)

Comprehensible proposal

Cum laudes

Elaborate proposal

Grants

International experience

Previous employersinstitutes

Publication record

Writing skills

Ability to work hard

Ability to work independently

Ambition

Authenticity

Enthusiasm

Goal directed

Leadership skills

Originality

Perseverance

Self-consciousness

Willingness to learn

Ability to motivate others

Being proactive

Being social

Communication skills

Fit in a group

Having a large network

Persuasiveness

Social skills

Team spirit

Different views on scholarly talent 5 of 12

Currently it is extremely important to enter the Veni- and Vidi-

trajectory or something similar at a very early stage From thebeginning of your career you should have received some sort ofmark indicating you are on the right track Those will get a plus

for sure they successfully passed the procedure And theselections are very heavy everyone knows There you donrsquothave a 80 chance but more likely the opposite of 10 to 20So if you have survived it gives you the mark lsquotoprsquo meaning you

are obviously very good (male NTS)

As the selection is tough those who managed to receivea grant are assumed to be excellent researchers In thesociology of science this phenomenon is also known asthe Matthew effect Scientific credits like awards andgrants are more often allocated to researchers whoreceived them before than to those who have not evenwhen the quality of their work is similar (Merton 1968) Interms of conversion of professional capital (grants) intosymbolic capital (prestige) grants have a very highexchange rate

As these personal career grants are seen as importantindicators of talentmdashalso within career policymdashwe will inthe next section take a closer look at this process of grantallocation

32 The concrete concept of talent within grantallocation

The process of grant allocation within the InnovationalResearch Incentives Scheme (Vernieuwingsimpuls)consists of several evaluation phases (see Fig 1)2 Thecriteria used to evaluate the applications are found tovary across the several evaluation phases For thisreason we successively describe the various phases andwhich criteria are primarily used by individual reviewersin each phase After subsequently comparing this concretecase to the just described general identification of talent onthe individual level we will describe how talent isdeliberated within panels and how panels reach agreementwith regard to talent selection

33 Preselection many applications limitedinformation

In case of a much higher number of applications comparedwith number of grants the allocation procedure involvespreselection as often occurs For the preselection panel-lists receive the research proposals and the curriculumvitae of the applicants Applications are not yet sent toexternal reviewers who can be considered to be theexperts in the specific research area Panels are composedin such a way that their members cover a broad expertiseThis implies that the panel as a whole is assumed to havethe expertise to review all proposals as good as possibleAt the same time this implies that a single panellist lacksknowledge to accurately review all proposals Intervieweesindicate to be aware of this and many of them consider the

broadness of the set of applications as problematic and

impeding the selection process Especially taking into

account that the rejection rate in this phase is often

about 40

You need to review and judge research and performance ofpeople from a completely different research field And within thesocial sciences these differences are really huge Certainly last

time we had people from psychology to law and so on And yesthe tradition of publishing and way of working everything is sodifferent that you really are comparing apples and oranges [ ]At the same time it is an enrichment as you are confronted with

things you consider to be normal while others donrsquot and the otherway around But it is quite difficult (female SSH)

Because of the heavy workload and the wide range of

topics panellists tend to primarily focus on the curriculum

vitae of the applicant The curriculum vitae proof of

mainly professional capital is found to be generally easy

to review for all applicants regardless of disciplinary

proximity

lsquoThe applicant can be evaluated rather objectively You can put alist next to it and count like how many publications how manyawards grants honor degrees that kind of thingsrsquo (female SSH)

List of publications is mentioned as the most important

indicator of talent in this first selection phase Secondly

Figure 1 Schematic overview selection procedure Innova-tional Research Incentives Scheme (VeniVidi)

6 of 12 P van Arensbergen et al

especially according to female interviewees there shouldbe a notification of international experience More gener-ally interviewees use the institute lab or organizationitself an applicant previously worked at as a qualitycriteria However this only holds for the NTS domainas in the SSH domain it was only mentioned onceSimilar to what we found with regard to general talentselection honour degrees awards and grants are highlyvalued as professional capital

In this first evaluation phase the emphasis is on profes-sional capital as three quarters of all criteria mentionedare related to this type of capital To a much smaller extentpanellists review individual and social capital These typesof capital are found harder to assess based on resumersquos andwritten proposals Nonetheless panellists do look for in-dications of authenticity independence and leadershipskills (individual capital) and for quality of the applicantrsquosnetwork (social capital) They mainly ground their assess-ment on (co-)authorship patterns People are positivelyevaluated when they published without their promoterwith various researchers from other (international) insti-tutes and on topics different from their PhD researchThey are seen as having clear goals and ideas about whatthey personally want to accomplish as being able to createtheir own niche and network (authenticity) and to workindependently

Although the applicantrsquos resume is found to be mostdecisive abstracts of research proposals are importanttoo All panellists regardless of specific expertise shouldbe able to generally understand the abstract If it is toospecific or too much jargon is used it will not be evaluatedpositively Applicants are expected to be able to inform abroad audience about their research ideas

When I started I remember in the first round I returned five or so

of the thirty applications I had to review I said I was not able toassess them They were far out of my working range In thesecond year you donrsquot do that anymore as for the third year

Then you may think oh no not another one you donrsquotunderstand But after all it is up to the applicants to makeclear to everyone who reads the abstract what they aim to do If

they succeed it is fine it is a good application If they fail it isnot a good application (female NTS)

In short in the preselection phase the emphasis clearly ison professional capital as assessed from resumes Besideshaving an impressive publication list including first-authorpublications independent from the promoter and on a dif-ferent topic talents are able to communicate their researchideas in a clear and generally comprehensive way

331 Involving external peer review bringing in moreexpertise Next panels have to select applicants for theinterview round Proposals are reviewed more thoroughlybeyond resume and abstract At first proposals are moreglobally assessed and panellists determine their level ofexpertise regarding the specific topic When one concludes

to have enough expertise (s)he will critically reviewthe proposal in more detail Otherwise (s)he will leavethe thorough evaluation of the actual content of theproposal up to external reviewers who are the experts in

the specific research area

In this phase of the process applications are sent to(generally two or three) external reviewers based on their

specific expertise In an earlier quantitative study on grantallocation within the same Dutch research council andfunding program the external reviews were found toplay a modest role in the selection process which are even-tually decisive (Van Arensbergen and Van den Besselaar

2012) This can be explained because panellists indicated tonot automatically take over these expert reviews butevaluate and weigh them They primarily want to under-stand how the external reviewers determined their scoreWhen external reviews lack a clear motivation or whenpanellists disagree they can decide not to take it into

account when formulating their own score

Some external reviews are very good some are very bad It is

also possible to put the external reviewer offside I agree We hadseveral situations in which we thought that the review did notcome up completely objectively This person is only trying to

slice down the applicant instead of giving serious feedback Youalso check if your opinion on the applications matches theopinion of the external reviewers (female SSH)

When panellists evaluate the quality of the researchproposal the focus is predominantly on elaboration Itshould contain a complete description of the research

idea context methodology relevance etc No gaps orany missing information As in the preselection generallyunderstandable phrasing is strongly valued Other import-ant criteria are originality and innovativeness of researchtopics and feasibility of proposals However some pro-posals describe very innovative research questions but

lack any clear evidence that the applicant will be able toanswer them (in time) These proposals score lowerFurthermore panellists want to be convinced that theproposal is really the applicantrsquos work It should be part(or the beginning) of their own line of research not of their

PhD supervisor Just as in the preselection phase authen-ticity is highly valued showing their personal contributionto the proposal and giving it their own clear signature

To summarize in this phase the emphasis is still on pro-fessional capital as the most important criteria used bypanellists are track record and a clearly elaborated andfeasible grant proposal Elements of individual capital

are also assessed to some extent Talents are those appli-cants who are creative innovative and most of allauthentic

332 Selection after the interview having been faceto face with applicants In the next stage a selection ofapplicants is reviewed in a face-to-face interview of about

Different views on scholarly talent 7 of 12

30min Applicants present their proposal and answerquestions of panellists In line with earlier research panel-lists indicate this is a very decisive phase in the selectionprocess (see also Van Arensbergen and Van den Besselaar2012) For some applicants it is a clear turning pointmeaning a considerable shift in their ranking

I noticed that even if you received excellent external reviews it

can still go wrong completely That is the most important role ofthe panel you assume that if one has real talent it will show inan interview of twenty minutes You ask a few questions and then

one has to think flexibly enough to respond with lsquoyes indeedwhen this wonrsquot work out I still have this or that theoryrsquo So ithas to be someone who thinks about research in a realistic andflexible way and does not only have excellent papers You notice

rather quickly in interviews who you really consider to be atalent not only on paper but as a real person (female NTS)

First of all talents are able to give a clear presentationwith a right balance of showing academic expertise andusing a generally understandable language at the sametime Comments from the external reviewers given tothem in the previous selection round need to be takenup or contested with a clear explanation Furthermorethe way in which applicants answer the questions isstrongly assessed For panels this serves as an opportunityto test several of the criteria applied previously to thewritten application predominantly authenticity wherethey could have received strong support from eg theirpromoter while writing the proposal reflecting on theproposal they fully have to do themselves during the inter-view This gives panels more room to assess individualcapital and their academic habitus as applicantsrsquoanswers better reflect personal skills and ideas

Persuasiveness and enthusiasm but also if you donrsquot have thatenthusiasm needs to be there [during the interview] otherwise

yoursquore out An extra check whether the person clearly compre-hends [the topic] and whether it is really his own thing and theproposal is not written by someone else You try to figure out ifthe person really is into it As for content not much happens

anymore that happened beforehand [ ] I believe it are thesethings They are often hard to prove they are intuitive (maleNTS)

As this male interviewee indicates during the interviewintuition comes into play Writing and presenting aproposal can be improved by training and panels canevaluate them with a certain extent of objectivityHowever especially in this phase of the evaluationprocess more subjective criteria related to individualcapital play an important role mainly enthusiasm perse-verance and ambition Because it is all about allocatingpersonal career grants panellists want to be convinced thatapplicants really want to and are able to conduct theresearch project Enthusiasm needs to be accompaniedby strong persuasiveness As this has to do with commu-nication skills this is linked to social capital Finally pan-ellists look at personality of applicants As there are only a

few grants to allocate they prefer allocating grants tosomeone they like instead of someone who appearsarrogant or unfriendly

To summarize also in this evaluation phase theemphasis is on professional capital Academic skillsevaluated earlier in the procedure are tested during theinterview Does the applicant really have the knowledgeand skills to conduct the proposed research projectFurthermore during the interview much more attentionis paid to individual and to a smaller extent also tosocial capital Face-to-face contact enables panels toevaluate the real person behind the proposal Talents arethose who succeed in convincing the panel of their authen-ticity enthusiasm ambition perseverance and excellentcommunication skills We found that for male panelliststhe shift in focus during the interview from professionaltowards individual capital is larger than for females Thesame holds true for the NTS domain compared to the SSHdomain

333 Tension between general and concrete talentidentification After asking the interviewees how theyassess applicants and applications in each phase of thereview process we also asked them to describe the toptalents they came across during their grant panel workIn Table 4 we listed the 10 characteristics that were men-tioned most often by the interviewees both in general (firstpart of study) and concrete assessments (in grant panels)with the most frequently named traits at the top

The main difference between the lists of traits is thevariation in types of capital that are valued in talentsWithin the top 10 of general evaluation all three typesof capital are represented with number of acquired

Table 4 Top 10 characteristics of talent in general and concrete talent

evaluationa

General evaluation Concrete grant evaluation

Being social (sc)b Publication record (pc)

Acquired grants as previous

recognition (pc)cElaboration research proposal (pc)

General comprehensiveness (pc)

Ability to work hard (ic)

Ambition (ic)

Publication record (pc)

International experience (pc)

Authenticity (ic)

Enthusiasm (ic)

Originality (ic)

Self-consciousness (ic)

Ability to work independently (ic)

Enthusiasm (ic)

Perseverance (ic)

Writing skills (pc) Ambition (ic)

Hot topic in research proposalBeing proactive (sc)

aCharacteristics within the same cell were mentioned just as often by the inter-

viewees and are therefore ordered alphabeticallybsc= social capital pc=professional capital ic= individual capitalcAlthough grants can be considered professional capital interviewees referred to

them more directly as symbolic capital previous acknowledgement of prestige

8 of 12 P van Arensbergen et al

grants being a very important indicator for talent Inconcrete grant evaluation the emphasis is on professionalcapital complemented with several types of individualcapital but no social capital

334 Deliberating talent within panels Next we willexamine the decision-making process at the group levelHow is talent discussed within panels and what eventuallydecides who receives formal recognition as being a talentBoth before and after the interview panellists meet todiscuss their evaluations and come to a final priorityranking During these meetings each proposal is generallyintroduced and commented upon by one or two panellistswho were instructed to prepare a more extensive reviewAfter this other panellists have the opportunity torespond give their opinion or ask questions

Panel discussions offer panellists the opportunity toexplain criteria specific to their discipline eg publicationpractices or research methods As mentioned before manypanellists conceive the broad scope of the total set of ap-plications in terms of topics and research fields as prob-lematic By exchanging information with the panellistswho are experts on the specific proposal panel meetingsenable them to adjust their evaluation accordingly Thesame holds true for evaluating the innovativeness feasibil-ity and relevance of proposals To accurately assess theseelements one needs to be informed of the current state ofthe art of the related research field

You know the journals from the field you publish in yourself

You know the status and reputation of those journals andmaybe of a few from outside your field but of many journals youdonrsquot know this We had interesting discussions about this inwhich an applicant was about to be put aside because he

lacked top publications when a panellist said lsquoBut this is THEjournal in this field so it wonrsquot get any better than thisrsquo (maleSSH)

In general the panel agrees with and follows the opinionof the field expert(s) within the panel Lamont (2009)described this as adhering to the rule of lsquodeferring to ex-pertisersquo Experts generally have affinity with the topicwhich can make them put extra effort in convincing theother panellists of the strengths of that application Buttheir knowledge can make them more critical too identify-ing more easily the weaknesses of applications Panellistsare also found to be quicker enthused by topics they do notknow much about Some fields of research seem to be moreattractive for non-experts because topics are more appeal-ing In these situations the experts sometimes consider ittheir task to temper this enthusiasm by eg explainingflaws in the research design

Compared to earth sciences people are generally easily enthused

[about life sciences] So often I say lsquowell this is not thatinterestingrsquo instead of lsquowhy donrsquot you understand that this is veryinterestingrsquo But yeah this certainly does not hold for earth

sciences They often had to convince us that it was very

interesting Those times I didnrsquot see this (male NTS)

With regard to top talents the majority of the inter-

viewees indicated the entire panel easily recognized them

and not much discussion is needed to determine the top

three3 Also the least impressive applicants are said to be

easily identified Most deliberation time is spent on the

large middle area in between Quality differencs within

this group are very small leading to a rather arbitrary

boundary between just-selected and just-rejected appli-

cants (see Van Arensbergen and Van den Besselaar 2012)

Actually only the top 2 is evident Of numbers 3 4 5 and so on I

have the feeling of yeah it could have been otherwise It is allbeing exactly calculated You have to assign scores includingdecimals This gives a kind of impression of quasi exactness but

it isnrsquot (female SSH)

The average scores within this group are almost the

same but applicants generally vary on different criteria

One may have a better track record but less international

experience or one gave a more convincing presentation

but has a less innovative research proposal Therefore it

is very important which criteria the panellists emphasize

during the discussion This is found to be dependent on

various factors like panel composition and the comments

made by the first speaker (see also Langfeldt 2002

Lamont 2009 Luukkonen 2012) The composition deter-

mines the available expertise but also more subjectively

the affinity with the research topic or methods within the

panel What the first speaker starts the discussion with is

also found to be decisive according to the interviewees

The strong or weak points of the applicant mentioned

first are strongly supported by other panellists

So when someone is assessed what the first reviewer states willset the tone When he completely tears down the proposal the

rest is only damage control When he praises it with argumentsthey others will say yes this is also a way to look at it and theywill immediately consider it as something positive (male

interdisciplinary)

Furthermore personal preferences and the atmosphere

within the panel are found to influence final selection de-

cisions too Although panellists are convinced that in the

end the top talents are granted they indicate to have

doubts about part of the allocation decisions Because

the quality differences are small and random (social)

factors influence decisions related to the middle group

many rejected applicants could have received grants as

well Most interviewees indicate they are aware that this

is inherent to the review process and they would not know

how to change this As the process of grant allocation is

conducted by human beings and involves social inter-

action these subjective factors can never be fully

excluded (Lamont 2009)

Different views on scholarly talent 9 of 12

335 Talent and gender In the interviews we deliber-ately did not ask explicitly for gender differences withregard to both panellists and applicants As gender ingeneral is quite a sensitive and highly debated topic wewanted to see whether interviewees would bring up thegender issue by themselves The large majority did notOnly a few times (mostly female) interviewees mentionedgender differences explicitly as in female candidates beingmore introvert and emotional males being more rationaland self-confident All other references to gender weremade to emphasize that gender was not an issue withinthe panel Interestingly these all came from female inter-viewees Women seem to be more conscious of the genderissue than men as the following quote also shows

I lsquoOkay I asked all my questions Is there anything you would

like to addrsquoR Yes there is one thing I was really wondering whether youwould ask me about Namely to what extent gender plays a rolein this whole process Just briefly in my mind it is a point of

attention I mean it is something Irsquom very attentive to MyselfSo I mean if things would happen of which I think well this isnot acceptable then that would be problematic for me But I

have to say Irsquom pleasantly surprised (female SSH)

The only difference we found between male and femaleinterviewees is that in describing top talents both as grantapplicants and as general university employees womenascribe a higher value to individual capital

4 Conclusion and implications

Analyzing talent in terms of symbolic capital this studycontributes to previous research on talent assessment orevaluation practices in several ways Firstly it showedthe dynamics of evaluation criteria in various phases ofthe grant allocation process not only with regard to thefinal outcomes We described how criteria relating tovarious types of capital and the weight assigned to themchange during the review process Secondly we showed thedifferences between what characteristics count as talentwhen (early career) researchers are selected by grantpanels and the more general notion of talent used by(senior) scholars

To start with the second issue the general description ofacademic talent reflects the various activities scholars areinvolved in and it entails a broad variety of skills andtraits combining professional social and predominantlyindividual capital Within the concrete context of grantallocation a much narrower definition of talent is usedand selection is mainly based on professional capitalCounts of publications awards grants and internationalexperience are key criteria used in the selection processespecially in the first phase Only later in the procedureindividual capital is added Being highly productive is ob-viously an important part of the academic habitus Morespecifically in case of ECGs as discussed in this article

being highly productive can be considered an importantentrance criterion

As grants are distributed with specific aimsmdasheg to doinnovative researchmdashit is not surprising that the notion oftalent differs between this concrete and the more generalforms of talent identification However it is important tocreate a better understanding of talent selection in grantpanels because of the considerable significance assigned togrants Grants more specifically the personal careergrants provide academics not only with financial resourcesto conduct research but also with academic prestige andfurther career opportunities The acquirement of careergrants like those of the Dutch Innovational ResearchIncentives Scheme is considered an important indicatorof talent Competition for these grants is strong andhaving succeeded in obtaining one is seen as proof thatthe grantee belongs to the top talents Actually being inthe top 10 talents in onersquos field is one of the formalevaluation criteria Many interviewees indicated the onlypossibility they saw for their talents to stay in academiawas by obtaining external grants Within recruitmentevaluation and promotion procedures acquirement ofexternal fundingmdashand especially prestigious careergrantsmdashform an important criterion Skills and dispos-itions needed to acquire grants have been included in theacademic habitus

When obtaining prestigious career grants becomesleading in recruitment the tension with the more generalnotion of talent becomes important There is the risk of defacto undervaluation of other aspects of individual andsocial capital needed for core tasks of university staff asteaching management and valorization The growing im-portance of research grants could also have negative con-sequences for risky and creative research as reviewprocedures adopted by funding agencies for allocatinggrants tend to contain a conservative bias As evaluationsare constrained by the boundaries of current knowledgeand reviewers generally highly value criteria like validityplausibility and a strong publication record researcherswho move to a new field or who have lsquowildrsquo ideas have lesschance being funded (Heinze et al 2009 Luukkonen2012) The question remains whether these lsquootherrsquo skillsare increasingly neglected or whether these come up inthe later part of eg grant selection processes Thisbrings us to the issue of changing characteristics of whatcounts as talent in grant selection procedures

In the second part of our study we studied the variousstages of panel reviewing and deliberation following up onresearch by Langfeldt (2002) Lamont (2009) andLuukkonen (2012) Talent proved to be assessed differ-ently across the various phases of the selection processand panels change the way they discuss applications andreach agreement Within every phase there is an overallagreement on which skills and traits applicants need tohave However panellists face two main difficulties firstthe broadness of the set of applications they have to review

10 of 12 P van Arensbergen et al

and second the minimal quality differences within themain group of applicants Because of the broadness ofthe applications under review panels have to cover awide area of research Therefore panellists can only beexperts to some of the applications making the panelusually a lsquopanel of generalistsrsquo (Luukkonen 2012)Discussions about individual applications are dominatedby the few lsquorealrsquo experts but all panellists are involved inevaluating the applications and in decision-making Incases where the expert panellists evaluate applicationshighly positively these scores will be averaged with theother scores of panellists who maybe less familiar withthe specific research area methods or subjectConsequently high potentials which are not recognizedas such by the majority within the panel will end up inthe so-called grey area in which average quality scoreshardly varymdasheven if the experts did recognize them

The second difficulty the panellists face is that within thelarge set of applicants no clear differentiation can be madeusing the most common criteria (eg publications grantsinternational experience) This is often clear in thefinal scores which hardly differ between many of the ap-plicants This implies that other criteria are involvedmdashexplicitly or implicitly These criteria cannot always befully articulated as they are often more subjective andrelated to the tacit dimension of evaluation (Bourdieu2004 Van den Brink 2009) As grant decision-making isa social process involving human interactions subjectivityand random factors can never be completely excludedthey are inherent to the evaluation process lsquoJudgingacademic excellence is a process shaped by real-world con-straintsrsquo (Lamont 2009 155) We showed how paneloutcomes are influenced by individualsrsquo behaviour (egpersonal preferences) context-specific characteristics(eg time restrictions) and group dynamics (eg influenceopening statement) Panel decisions convert minor differ-ences in quality into enlarged differences in recognition

Owing to these difficulties many panellists indicate toquestion the final allocation decisions Even though theyare convinced the top talents are amongst the grantees alarge share of the grantees are considered not to be anybetter than many of the rejected applicants Many panel-lists indicate that decisions with regard to this share areliable to arbitrary factors and are partly a matter of luckConsequently one would expect a devaluation of thesymbolic value of these grants Paradoxically despiteacknowledging this level of subjectivity and arbitrarinessexperienced researchers still ascribe high symbolic value tothese grants

Further research is needed to determine the actual valueof these grants for early career researchers Following upon studies by Laudel and Glaser (2012) Gerritsen et al(2013) and Bloch et al (2014) it is necessary to betterunderstand whether and how these grants enhanceacademic careers and moreover whether these researchersare more successful than those without such grants

Observational studies adding to the interesting work ofLangfeldt (2002) and Lamont (2009) would createopportunities to further identify the exact review criteriaincluding the more tacit criteria used within panels andthe effects of group dynamics on the outcomes of panelreview processes By making these implicit criteria explicitone may be able to avoid that applicants are assessed ondifferent criteria and may reduce the influence of subject-ivity and random factors when selecting among applicantswith almost equally good publication records And towhat extent do these implicit criteria cover the dimensionsof the more general concept of talent that resulted fromthis study

Acknowledgements

The authors would like to thank three anonymous refereesfor their comments on an earlier version of this article

Notes

1 For an indicator based on similar ideas Van denBesselaar et al (2012)

2 See Van Arensbergen amp Van den Besselaar (2012) formore details on the allocation procedure

3 This could not be confirmed in an earlier study ontalent selection in grant panels (Van den Besselaar ampVan Arensbergen 2013) In most panels no clear topcould be discerned based on panel review scores espe-cially not in the beginning of the selection process

References

Baron-Cohen S (1998) lsquoSuperiority on the Embedded FiguresTest in Autism and in Normal Males Evidence of an ldquoInnateTalentrdquorsquo Behavioral and Brain Sciences 213 408ndash9

Bleijenbergh I Benschop Y and Vennix J (2013) lsquoGroupModel Building to Support Gender Equality ChangersquoIn Achterbergh J Benschop Y Hendriks P and Van deVen A (eds) Op zoek naar het andere een liber amicorumvoor Hans Doorewaard pp 80ndash96 Boom Meppel

Bloch C Graversen EK and Pedersen HS (2014)lsquoCompetitive Research Grants and Their Impact on CareerPerformancersquo Minerva 52 77ndash96

Bourdieu P (1986) lsquoThe forms of Capitalrsquo In Richardson J(ed) Handbook of Theory and Research for the Sociology ofEducation 1st edn pp 241ndash58 New York Greenwood

mdashmdash (1988) Homo Academicus Stanford Stanford UniversityPress

mdashmdash (2004) Science of Science and Reflexivity Cambridge TheUniversity of Chicago and Polity Press

Brass D J Joseph G Greve H R and Tsai W P (2004)lsquoTaking Stock of Networks and Organizationsrsquo Academy ofManagement Journal 476 795ndash817

De Boer H and Jongbloed B (2010) Analyse van universitairejaarverslagen 2009 Achtergrondstudie voor RathenauInstituut Enschede CHEPS

De Jonge Akademie (2010) Rendement van TalentAanbevelingen voor Motiverend en StimulerendLoopbaanbeleid Amsterdam De Jonge Akademie

Different views on scholarly talent 11 of 12

Gerritsen S Plug E and Van der Wiel K (2013) Up or outHow Individual Research Grants Affect Academic Careers inthe Netherlands (No 249) Den Haag CPB NetherlandsBureau for Economic Policy Analysis

Heinze T Shapira P Rogers J D and Senker J M (2009)lsquoOrganizational and Institutional Influences on Creativity inScientific Researchrsquo Research Policy 38 610ndash23

Howe M J A Davidson J W and Sloboda J A (1998)lsquoInnate Talents Reality or Mythrsquo Behavioral and BrainSciences 213 399ndash407

Lamont M (2009) How Professors Think Inside the CuriousWorld of Academic Judgment Cambridge HarvardUniversity Press

Langfeldt L (2002) Decision-Making in Expert PanelsEvaluating Research Constraints Processes and Bias OsloNorwegian Institute for Studies in Innovation Researchand Education

Laudel G and Glaser J (2012) lsquoThe ERCrsquos Impact on theGranteesrsquo Research and their Careers EURECIAUnderstanding amp Assessing the Impact amp Outcomes of theERC Funding Schemesrsquo Work Package 4 Summary Reportlthttpwwweurecia-ercorgwp-contentuploadsEURECIA-ImpactOnResearchContentAndCareers-SummaryReportpdfgt

Lepori B Van den Besselaar P Dinges M Potılsquo B RealeE Slipersaeter S Theves J and Van der Meulen B (2007)lsquoIndicators for Comparative Analysis of Public ProjectFunding Concepts Implementation and EvaluationrsquoResearch Evaluation 164 243ndash55

Luukkonen T (2012) lsquoConservatism and Risk-Taking in PeerReview Emerging ERC Practicesrsquo Research Evaluation 2148ndash60

Merton R (1968) lsquoThe Matthew Effect in Science The rewardand communication systems of science are consideredrsquoScience 1593810 56ndash63

Neufeld J Huber N and Wegner A (2013) lsquoPeer Review-Based Selection Decision in Individual Research FundingApplicantsrsquo Publication Strategies and Performance TheCase of the ERC Starting Grantsrsquo Research Evaluation 224 1ndash11

Olbrecht M and Bornmann L (2010) lsquoPanel Peer Review ofGrant Applications What Do We Know from Research inSocial Psychology on Judgment and Decision-Making inGroupsrsquo Research Evaluation 194 293ndash304

Thomas D and Nedeva M (2012) lsquoCharacterizing Researchersto Study Research Funding Agency Impacts The Case ofEuropean Research Councilrsquos Starting Grantsrsquo ResearchEvaluation 21 257ndash69

Thunnissen M Fruytier B and Van den Brink M (2010)Beleid en Beleving Onderzoek naar Jongetalentenbeleid opNederlandse Universiteiten Den Haag SoFoKleS

Van Arensbergen P (2014) lsquoAcademic Talent Selection inGrant Review Panelsrsquo In Prpic K Van der Weijden Iand Aseulova N (eds) (Re)searching Scientific Careers StPetersburg IHSTRAS amp SSTNETESA

Van Arensbergen P and Van den Besselaar P (2012) lsquoTheSelection of Scientific Talent in the Allocation of ResearchGrantsrsquo Higher Education Policy 25 381ndash405

Van Arensbergen P Hessels L K and Van der Meulen B(2013) Talent Centraal Ontwikkeling en selectie vanWetenschappers in Nederland Den Haag Rathenau Instituut

Van den Besselaar P Sandstrom U and Van der Weijden I(2012) lsquoThe Independence Indicatorrsquo In Archambault EGingras Y and Lariviere V (eds) Science amp TechnologyIndicators 2012 pp 131ndash41 Montreal OST amp ScienceMetrix

Van den Besselaar P and Van Arensbergen P (2013) lsquoTalentSelection and Funding of Researchrsquo Higher Education Policy26 421ndash7

Van den Brink M (2009) Behind the Scenes of Science GenderPractices in the Recruitment and Selection of Professors in theNetherlands PhD thesis Radboud Universiteit Nijmegen

Van den Brink M Fruytier B and Thunnissen M (2012)lsquoTalent Management in Academia Performance Systemsand HRM Policiesrsquo Human Resource Management Journal232 180ndash95

Waring M (2013) lsquoAll in this Together HRM and theIndividualisation of the Academic Workerrsquo HigherEducation Policy 26 397ndash419

12 of 12 P van Arensbergen et al