A review of the Value for Money assessment of Science Foundation Ireland

Ali BasakRichard O'Carroll

MSc EPS Year 2EC8011 – Assignment 2

Background

Science Foundation Ireland (SFI) invests in academic researchers and research teams who are most likely to generate new knowledge, leading edge technologies and competitive enterprises in the fields of science and engineering underpinning three broad areas: Biotechnology, Information and communications technology, Sustainable energy and energy-efficient technologies

Activities previously reviewed by Brook in 2005

This evaluation completed by Indecon in 2008

Assessment takes place in context of the Government’s Value for Money and Policy Review Initiative  and looks at the SFI’s

programmes under three headings:

Terms of reference

Objectives

Impacts

Complentarities/Linkages

1. Examine validity of programmes in meeting the longterm objectives of SPI to create highly visible critical mass of word-class rearch excellence

2. Examine validity of programmes in context of changing circumstances and overall consistency with overall objectives of the SSTI, other relevant policies and EU strategies in the area

3. Determine probable outcomes that could result from discontinuation, reduction or expansion of any programme

4. Review individual objectives of each programme and assess extent to which they are being met

5. Examine separate programmes in terms of comprehensive measurement of inputs and outputs

1. Review effectiveness of operation of individual programmes in terms of outputs and impacts where possible. Also quality of programmes and possible alternatives to them.

2. Review user satisfaction with management and implementation of programmes by SFI

1. Review effectiveness of operation of individual programmes in terms of outputs and impacts where possible. Also quality of programmes and possible alternatives to them2. Review user satisfaction with management and implementation of programmes by SFI

Methodology

Evaluation of programme outputs

Review of research outputs

Review ofCollaboration

Activity

Review ofCommercialisation

Activity

Outputs of journal-based peer review articles published by SFI funded researchers

Outputs in relation to international conference presentations or papers delivered by SFI-funded researchersInternational collaborations

Industry collaborations

Views of industry on SFI objectives

Data considered: SFI commitments & expenditure; Research activities & outputs; Consultation of stakeholders/experts & surveys

Programme effectiveness & impact

Assessment of wider economic impacts

Programme Management & Monitoring

International Comparative

Position of SFI

Assessment of research outputs

& quality

Early citation rates

“International” quality of outputs based on appearance in peer-reviewed journals

“Productivity" of researchers

Programme effectiveness & impact

Assessment of wider economic impacts

Programme Management & Monitoring

International Comparative

Position of SFI

Assessment of research outputs

& quality

Economic benefits of investment in human capital

Inward Investment effects – based on discussions with the IDA and a reference to three partnerships between multinationals and Irish third level colleges

Commercialisation i.e. actual revenues

Programme effectiveness & impact

Assessment of wider economic impacts

Programme Management & Monitoring

International Comparative

Position of SFI

Assessment of research outputs

& quality

Discussion of peer review process

User satisfaction with management and implementation based on an extensive survey – note survey response ratios

Discussion of approaches to monitoring

Programme effectiveness & impact

Assessment of wider economic impacts

Programme Management & Monitoring

International Comparative

Position of SFI

Assessment of research outputs

& quality

Based on discussions with a list of international academics

The concept of “virtual citations”

Conclusions

• The report concludes that a discontinuation of the programmes would adversely affect the buildup of research capacity

• It observes that SFI has unusually high staff turnover

• It points out room for improvement in interaction with other agencies, in particular the HEA

• It notes that 1.4 billion Euro spent so far and that VFM and financial sustainability will be of increasing importance

Recommendations

Issues

Some independent or disinterested input would

be useful. Maybe some comparisons with

international or European benchmarks for countries of similar size would be an idea. But sure that’d

be just mad, Ted.

There has been an increase in PhDs but there is no clarity about the counterfactual or control methods.

No return on investment can be expected in the short run. This is assessed by recipients of SFI funding only.

No serious international bench-marking. Only comparator is Irish programmes not funded by SFI. “Virtual impact” particularly worrying because of this.

Methodology merely repeats the objectives so not clear how objectives of review will be met or that they have been.