21
Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna 15 and 16 December 2008

Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Embed Size (px)

Citation preview

Page 1: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Relevance and Impact of

Humanities Research

and how ERiC can help

Jack SpaapenRoyal Netherlands Academy of Arts and Sciences

Humanities Conference Vienna 15 and 16 December 2008

Page 2: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Relevance, impact and evaluation

Various functions of evaluationVarious functions of evaluation Judgment about Quality, both scientific and societal Accountability to university, to government, to society Improvement of research quality and impact

Various problemsVarious problems

Functions get mixed in evaluations, but with different weight Some seem to be more obvious than others Criteria and indicators slanted towards scientific quality and to practices of natural and biomedical sciences Different actors have different expectations

Peer review under pressure, evaluation procedures tooPeer review under pressure, evaluation procedures too

Page 3: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Evaluating Research in Context (www.eric-project.nl)

endorsed byendorsed by * Dutch Academy, * National research council (NWO), * Association of Dutch universities, * Council of the professional schools,* Rathenau Institute (Science Systems assessment) * QANU, * Ministry of Education and Sciences

Page 4: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

ERiC: Development of alternative evaluation methods

WHY ERiCWHY ERiC

to inform boards / government about societal relevance of research (accountability and Lisbon ambitions)

to help research areas that are not happy with dominant evaluation approaches : humanities, social sciences, technical sciences, health research, MIT, etc.

Page 5: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Core problem

The world is changing

Science (academic) is changing into research (MIT) (Latour)

Evaluation is not yet changing, or at least still dominated by criteria and indicators that used to work in the old situation

Page 6: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

methodological development

RMW: the societal impact of health research [2002] SWR / RGW: judging research on its merits [2005] ERiC: evaluating research in context [2005, 2007] AWT: alfa stralen (humanities shine) disciplinary developments: health research (RGO /

UMCs), development research (CERES); law (disciplinary committee)

pay back, UK research councils, Scandinavian countries, etc.

examples from outside Europe: USA, Australia, Canada

Page 7: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

“Judging research on its merits”[KNAW/SWR/RGW 2005]

Collaboration of two Academy Councils, social sciences and humanities

Dissatisfied with current evaluation systems mainly focusing on SCI (high impact journals)

Two problems: 1. doesn’t necessarily fit research communication patterns in soc sc and hum; 2. doesn’t value societal output which is important for a lot of fields

Search for more fitting evaluation schemes

Page 8: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Necessary conditions for better evaluations

Self evaluation focus on both scientific reputation

and communication, and also on interaction with other audiences

List of target groups: peers, students, professionals, policy makers, business, broader public

List of indicators per target group: publications, citations, but also text books, reviews, grants from policy, collaborations with business, professionals, awards, popular publications, etc.

Benchmarking is critical process

Page 9: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Goals of ERiC

stimulate debate about alternative methods of evaluation, fit for all disciplines regarding the value of research for society

develop methods to do such evaluations

website www.eric-project.nl

Page 10: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

how does ERiC work?

Context group as a central unit publications guidelines workshops, national and international pilot studies : agricultural research,

pharmaceutical research, architecture, law, engineering

European project : SIAMPI website : www.eric-project.nl

Page 11: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Results so far

Working definition of Societal value:o As a broad concept : societal relevanceo As a target-oriented concept : societal impacto As a mainly economic concept : valorization

these three have different consequences for evaluation

Assessment approach fit for all disciplines: based on an analysis of the mission orientation of a

research group o scientific communityo professional sectoro Industryo policy-orientedo society at large

Test method

Page 12: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Test model: 4 steps

1. Reflection on mission orientation, self evaluation report

2. Empirical stage, find criteria and indicators for quality and relevance

3. Consultation with researchers and stakeholders, presentation of findings

4. Feed back and forward look

Page 13: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

Step 2 : productive interactions

Identify social domains in context (policy, industry, society at large)

Analyse the 4 main interaction channels between research and context: texts, people, artifacts, money

Feed back to research community and stakeholders [workshops]

Page 14: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

example of evaluation of societal quality – radar graph

0

0,5

1

1,51

2

3

4

5

6

789

10

11

12

13

14

15

example3

Science

Market

Societa

Page 15: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

example REPP – table graph

Science, certified knowledge  

relative citation impact --

productivity scientific publications+

+**(1

international visibility and collaborations =

representation in editorial boards ++

invited lectures ++

Industry, market  

non-academic/commercial citing environment ++

productivity professional publications+

+*(1

involvement in industry/market -

advisory and expert roles in commercial domain --

editorships professional journal ++**

Policy, societal  

involvement in policy domain +

memberships and expert roles in governmental bodies ++

memberships of societal organisations: advisory/ education ++*

production of public goods +

additional grants from policy +

Page 16: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

example of evaluation of societal quality – radar graph

Crop and Grassland Science

% co-publ. Neth. (not KCW)% Neth. citing (not KCW)

% co-pub. Internat.

% Internat. citing

% cited journ. art.

2 cit. / journ. art.

2 journ. art. / fte total

1,5 member sci journal/ fte WP

orientation on science ( in %)

second fte (20 % of fte total)

% coop/fin proj ->res. grps (incl.KCW)

% mobility to research (incl. KCW)

diss. (cat 1)/ 4 fte AIO/OIOJunior, AIO,OIO students (1 Jun. / 2 Sen. -

1997))# diss (cat 1) / # junior staff% mobility to company

third budget fte (45% of fte total)orientation on professionals ( in %)

coop/fin proj innovating inst

1 patent / 3 WP

2 prof art/ fte total

1,5 member advisory board/WP

% coop/fin proj gov/spec. prog.

1,5 members gov. or spec. prog./WP

% mobility to gov./policy arena

involvement NGO's (scored in %)

% mobility to KCW

KCW coop/fin proj

KCW co-pubKCW citing

Research Embeddement & Performance Profile

Science & Certified Knowledge

Collaboration & Visibility

Public Policy

Innovation & Professional Education

& Training

KCW

2,1 memb / fte senior2,9 journ art / fte tot

0,54 f te junior / 1 fte senior

extra 15,25 dissertations (cat 1)

Page 17: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

ERiC and the humanities

Express the relations of humanities research with society in a meaningful way

Help find criteria and indicators that might help to evaluate that

Bring together the expertise of both stakeholders and researchers

Stimulate them to find consensus about how to evaluate

Page 18: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

What can humanities do?

don’t try to emulate other fields try and find consensus as a field about

indicators convince policy makers and managers

that you are capable of defining indicators that are robust and fit for your field

Page 19: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

NetherlandsFrance, NL, EUNL, UKSpain, UK

HealthNanotechnologyICTSocial sciences and humanities

ERiCSIAMPI

www.ERiC-project.nl

Page 20: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

focus in human sciences

1. Describe the content, (societal) objectives, organization and evaluation practices of the program/projects involved in the pilot.

2. Identify research strategies and policies aiming at measuring the scientific and social impact of the program/projects in the various relevant dimensions.

3. Discuss the findings with the stakeholders related to this case, evaluating the validity, reliability and relevance of the results.

4. Advise on further development of methods for the assessment of social impact. Analysis of strength, weaknesses and possible improvements for the method to be developed.

Page 21: Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

ERiC and SIAMPI and the Humanities

1. ERiC : cases in the Netherlands, but perhaps abroad

2. SIAMPI : dissemination of results through comments, conferences, joint workshops