20 YEARS OF SCIENTIFIC RESEARCH IN 20 YEARS OF SCIENTIFIC RESEARCH IN HEALTH/WORK/ENVIRONMENTHEALTH/WORK/ENVIRONMENT

September 6, 2012 September 6, 2012

Thoughts of a reviewerThoughts of a reviewer

Prof Dick Heederik, PhDIRAS, Utrecht University, The Netherlands

Reviewing scientific proposals Reviewing scientific proposals and programsand programs

Involved in BESLPO project evaluation and the SSD Health program evaluation as panel member

Program evaluation SSD Program evaluation SSD HealthHealth

Scientific quality Networking Internationalization Policy relevance Coverage of the program Characteristics of the program

Project level (network, budget, duration …) Program level (calls, budget, …) Follow-up committee International projects/EU Clusters Dissemination

Relevance of the program Other considerations

Developments in the scientific Developments in the scientific community community

Scientific production more dominated by teams, even in field traditionally dominated by solists

Teams produce more highly cited papers Development is seen in all areas, over time,

even after removal of self-citations Networks have become the dominant and most

prominent way to go Research at disciplinary frontiers and in novel

areas is often inter-disciplinary Research management becomes interested in

R&D structures

The role of peer review in project The role of peer review in project and proposal selection?and proposal selection?

http://www.bishop-hill.net/

Criteria voor Quality Assessment Criteria voor Quality Assessment in Peer Review (NIH)in Peer Review (NIH)

Significance impact (does the project address an important problem or critical barrier to progress in the field)?

Investigators (well suited to the project)

Innovation (shift current research or practice paradigms)

Approach appropriate? Will the scientific environment

contribute to succes?

Scientific Quality: publications, Scientific Quality: publications, citations, publication networks …..citations, publication networks …..

From intuitive interpretation to quantitative analysis …

Quality: characteristics of good Quality: characteristics of good research groupsresearch groups

Leaders of high performing research groups survey: High performance research (publications,

citations (normalized for group size) Stronger research commitment More effort in group management Spent more time on network management All rounders

Verbree et al., Rathenau institute, NL

Quality: different types of Quality: different types of excellent groupsexcellent groups

Output types correlate poorly: publications, citations, productivity, citations per publication

…, and have different determinants. So, it also depends to some extent

on what is asked

Verbree et al. 2012 Rathenau Institute

Peer review program evaluation: Peer review program evaluation: output evaluation parametersoutput evaluation parameters

Program performanceProgram performance~1.8 Meuro/year internal support, ~ 70% external projects

Impact in different sub-fields Impact in different sub-fields

SSDSSD

Too early to make a formal quantitative analysis of impact of the BELSPO Health program

Does this result in unbiased impression given the likely additional funding from other sources?

In essence evaluation of participating groups

Networking and Networking and internationalizationinternationalization

Strong interdisciplinary collaboration (PARHEALTH, S2Nano, SHAPES)

Projects did not make use of additional funding possiblities to finance international partners

Some groups had strong international networks but connection with international research community could be strengthened

Collaboration with industry limited (S2Nano)

More formal approaches to analyze networks: 44-cluster co-authorship network of papers at the 10% highly-cited threshold (Rosas et al.

PLoSone, 2011 )

Collaborative outputCollaborative output

Dissemination Dissemination

Follow-up committee not for all projects useful, for others effective

Projects which have a stronger basic research focus could benefit from a scientific steering committee

More options for dissemination should be considered (internet databases, software tools, etc.)

To make scientific results available for society may require an additional research cycle

DisseminationDissemination

Role in evaluation of future exposure standards (PARHEALTH)

Results can be used by local planners (cost benefits of various modes of transport SHAPES)

Use of developed concepts in testing guidelines (S2Nano)

Breakthrough technology (ANIMO)

Coverage of the fieldCoverage of the field 20% of diseases associated with environmental factors

.. (Kirsh-Volders et.al. 2012) Occupational exposures (chemical, biological,

physical) Environmental exposures (outdoor, indoor) Do we know the priorities in our field (risk, impact, time,

DALY)?

The field environment and healthThe field environment and health

Small populations at risk, high risks (MICATR) Large populations at risk, low risks (SHAPES,

PARHEALTH) New emerging risks (S2Nano) New approaches/technologies (MIC-ATR, ANIMO)

Overall appreciationOverall appreciation

Small program Relevant for capacity building in

Belgium Relevant for public health in relation

to the environment in Belgium The program delivers value for

money Effect of most projects is beyond the

project period

Environment and health: fundingEnvironment and health: funding

Public health Public funding versus industry funding Mixed funding (Health Effects Institute)?

Where are we going?Where are we going?