Upload
tracy-allen
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Project Evaluation under the Project Evaluation under the 2007-2013 Convergence and 2007-2013 Convergence and
Competitiveness Competitiveness ProgrammesProgrammes
Richard GauntRichard GauntMark BeynonMark Beynon
CRG Research Ltd.CRG Research Ltd.
OverviewOverview
About CRGAbout CRG WEFO RequirementsWEFO Requirements Experiences and Perceptions of EvaluationExperiences and Perceptions of Evaluation Case StudiesCase Studies Evaluation – Principles and PracticeEvaluation – Principles and Practice Do’s and Don’tsDo’s and Don’ts Q & AQ & A
About CRGAbout CRG
Leading provider of research, evaluation, consultancy and policy development services
Completed over 400 projects for a wide range of clients in both the public and private sectors
15 dedicated researchers from a range of disciplines, capable of utilising a full range of qualitative and quantitative research techniques
High level of expertise in different areas of Welsh policy and funding, including EU policy and Structural Fund Programmes
Excellent data collection, entry and analysis services
Full bilingual (English/Welsh) service provision
ISO 9001 accredited procedures and quality assurance.
WEFO RequirementsWEFO Requirements
2007-2013 Programmes – all sponsors required to undertake or 2007-2013 Programmes – all sponsors required to undertake or commission evaluation of projectscommission evaluation of projects
All projects awarded £2million + in grant (ERDF/ESF) are All projects awarded £2million + in grant (ERDF/ESF) are required to have their projects evaluated by an external required to have their projects evaluated by an external independent contractor (+ projects involved in implementing independent contractor (+ projects involved in implementing ERDF innovative or experimental actions and innovative ERDF innovative or experimental actions and innovative projects under Article 7 ESF)projects under Article 7 ESF)
Projects below this threshold may still wish to appoint external Projects below this threshold may still wish to appoint external contractor because of independent perspective which can be contractor because of independent perspective which can be providedprovided
The size of the evaluation should be proportionate to the size The size of the evaluation should be proportionate to the size and complexity of the projectand complexity of the project
Costs associated with undertaking evaluation are eligible for Costs associated with undertaking evaluation are eligible for Structural Fund assistanceStructural Fund assistance
RationaleRationaleWhy EvaluateWhy Evaluate
Assess whether projects achieve their objectives How efficiently were outputs and results achieved Wider consideration of outcomes and impacts What would have happened without the intervention
Evaluation is wider in scope than monitoringEvaluation is wider in scope than monitoring Whilst monitoring data can help asses performance against objectives –
wont produce rounded judgement of success Evaluation – considers quality of achievements and contextual factors
which have helped/hindered success Example: Monitoring data will tell you 100 jobs created, but evaluation
will tell you the quality of those jobs, the impact on beneficiaries, and considers whether a proportion of the jobs would have been created in any case
Share good practice, make recommendations for future action.
Evaluation PlanningEvaluation Planning
Setting Aims and ObjectivesSetting Aims and Objectives Evaluation Plan to be submitted when developing your project
Outline evaluation activity during the life of your project
State when evaluation will be undertaken, by whom, and which evaluation methods will be used.
Plan will identify key evaluation questions, and the scope and size of the evaluation
Size of evaluation proportionate to size and risk of project – agreed with the WEFO Project Development Officer at development stage
Guidance on evaluation questions and suggested methodologies available on WEFO Website
This is an effectively an overview document – doesn’t necessarily have all the detail you need to design and implement the evaluation and ensure it is integrated into your project
Who should evaluate?Who should evaluate?
Size of project: > £2million = External, < £2Million = option to run it Size of project: > £2million = External, < £2Million = option to run it internallyinternally
If your project falls below threshold you may still wish to go external. If your project falls below threshold you may still wish to go external. Issues to consider include:Issues to consider include:
Do you have the skills internallyDo you have the skills internally Do you have enough staff resource to carry out the workDo you have enough staff resource to carry out the work Independent PerspectiveIndependent Perspective External Evaluation CostExternal Evaluation Cost Contract Management TimeContract Management Time
The other factor to consider is risk. High Risk/ High Profile projects will The other factor to consider is risk. High Risk/ High Profile projects will require a thorough evaluation which may well be resource intensive. require a thorough evaluation which may well be resource intensive. Project which are innovative or pilot in nature or where there are learning Project which are innovative or pilot in nature or where there are learning or participatory elements will need more resource intensive evaluations.or participatory elements will need more resource intensive evaluations.
Key Factors: Cost, Complexity, Resources, External PerspectiveKey Factors: Cost, Complexity, Resources, External Perspective
Objectives of Project EvaluationObjectives of Project Evaluation
Demonstrating the need for your project - feasibility
Evaluating Project Processes
Evaluating Project Outcomes
Reporting against Impact Indicators
Issues to ConsiderIssues to Consider
Formative vs. Summative – important decision that will shape the research Formative vs. Summative – important decision that will shape the research methodology and impact upon the way the project makes use of evaluation data. Big methodology and impact upon the way the project makes use of evaluation data. Big advantage of formative is that it provides an ongoing assessment of progress against advantage of formative is that it provides an ongoing assessment of progress against objectives, and allows opportunities/ corrective actions to be identified at every objectives, and allows opportunities/ corrective actions to be identified at every stage. stage.
Methods for collecting Data – Covers primary and secondary data, and methods Methods for collecting Data – Covers primary and secondary data, and methods should be appropriate to answer the key research questions. Selection of method has should be appropriate to answer the key research questions. Selection of method has resource implications. Weigh up the pro’s and con’s of different approachesresource implications. Weigh up the pro’s and con’s of different approaches
Methods for analysing data – should be considered when deciding the methods, size Methods for analysing data – should be considered when deciding the methods, size and scope of your evaluation. The method of data collection will have a clear impact and scope of your evaluation. The method of data collection will have a clear impact on the method of data analysis. The two main methods are:on the method of data analysis. The two main methods are:
Quantitative – statistics used to provide characteristics of a sample, and if there is sufficient Quantitative – statistics used to provide characteristics of a sample, and if there is sufficient data, to derive conclusions about performance of the projectdata, to derive conclusions about performance of the project
Qualitative – patterns and themes emerging from interview transcripts, data or recordings Qualitative – patterns and themes emerging from interview transcripts, data or recordings can be categorized to provide an analytical description of a sample and the key can be categorized to provide an analytical description of a sample and the key themes/issues emerging from a projectsthemes/issues emerging from a projects
Reporting & Dissemination – how do you intend to use evaluation data? Is it just to Reporting & Dissemination – how do you intend to use evaluation data? Is it just to satisfy WEFO requirements or will it actively be used as part of the project satisfy WEFO requirements or will it actively be used as part of the project management process.management process.
Research SpecificationResearch Specification
The Specification will set the parameters for the The Specification will set the parameters for the evaluation. It will need to include:evaluation. It will need to include:
IntroductionIntroduction Background/ Requirements Background/ Requirements Method/Budget – 2 options: detail method or detail budgetMethod/Budget – 2 options: detail method or detail budget Reporting/ DeliverablesReporting/ Deliverables DisseminationDissemination TimetableTimetable Data Protection IssuesData Protection Issues
Contract Award CriteriaContract Award Criteria Contract Management ArrangementsContract Management Arrangements CostCost
Internal/External?Internal/External?
Commissioning EvaluationCommissioning EvaluationIf external evaluators are to be used, need to If external evaluators are to be used, need to commission effectively. Issues to consider:commission effectively. Issues to consider:
Detailed Research Specification and Invitation to Tender Detailed Research Specification and Invitation to Tender Evaluation should meet project evaluation needs and WEFO requirementsEvaluation should meet project evaluation needs and WEFO requirements Need to comply with your own procurement guidelinesNeed to comply with your own procurement guidelines WEFO will advise on approach, method and budgetWEFO will advise on approach, method and budget PDO’s don’t appear to be paying much attention to evaluation pre-PDO’s don’t appear to be paying much attention to evaluation pre-
approval. Uncertainty over to what degree WEFO will involve themselves approval. Uncertainty over to what degree WEFO will involve themselves in the commissioning process at later stagesin the commissioning process at later stages
Evaluation costs are eligible for Structural Fund Support, so the Evaluation costs are eligible for Structural Fund Support, so the expenditure will be subject to approval, audit, monitoring, etc.expenditure will be subject to approval, audit, monitoring, etc.
Possibility of Supplier Frameworks being used to provide details of Possibility of Supplier Frameworks being used to provide details of approved ‘reputable’ companiesapproved ‘reputable’ companies
Decide on approach early and work with WEFO to ensure they are aware, Decide on approach early and work with WEFO to ensure they are aware, and happy with the way the project is being evaluated.and happy with the way the project is being evaluated.
The Evaluation ExperienceThe Evaluation Experience
Evaluation can be a powerful project management and review tool. Can measure progress against targets, identify what works well/ less well, provides the opportunity for corrective action to minimise risks and take advantage of unforeseen opportunities.
It also identifies best practice, and explores outputs, outcomes and impacts. Recommendations can inform both existing and future delivery.
If badly designed, poorly resourced, or ineffectively delivered they can be a burden to project managers, partners and beneficiaries.
Keen to hear your experiences?? Any good news/ horror stories? Any concerns about evaluation?
Case Study 1 Case Study 1 Research Skills TrainingResearch Skills Training
Objective 1 ESF funded collaborative project between Objective 1 ESF funded collaborative project between Bangor, Aberystwyth and Swansea UniversitiesBangor, Aberystwyth and Swansea Universities
Funded 67 PhD Studentship and ran collaborative Funded 67 PhD Studentship and ran collaborative research projects with regional SMEsresearch projects with regional SMEs
““to provide research training to enable individuals to to provide research training to enable individuals to develop the skills to contribute to research as develop the skills to contribute to research as professionals. The project will also increase the research professionals. The project will also increase the research capacity of SMEs, encourage them to undertake research capacity of SMEs, encourage them to undertake research and recruit researchers. It will also support the and recruit researchers. It will also support the development of key technology clusters in the region”.development of key technology clusters in the region”.
Case Study 1Case Study 1Research Skills TrainingResearch Skills Training
Summative EvaluationSummative Evaluation
Research Methodology:Research Methodology: Desk researchDesk research
Face to Face Interviews with academic supervisorsFace to Face Interviews with academic supervisors
Case Studies and Interviews with company supervisorsCase Studies and Interviews with company supervisors
Interviews with key management and administrative staffInterviews with key management and administrative staff
Focus Groups and telephone interviews with beneficiary Focus Groups and telephone interviews with beneficiary PhD StudentsPhD Students
Regular updates, draft final and final reports to clientRegular updates, draft final and final reports to client
Case Study 1 Case Study 1 Research Skills TrainingResearch Skills Training
Outcomes for the client:Outcomes for the client:
Thorough evaluation of the project model, internal processes, outputs Thorough evaluation of the project model, internal processes, outputs and outcomes for all parties – University, companies and most and outcomes for all parties – University, companies and most importantly the studentsimportantly the students
Highlighted successes, and explored why some collaborations failed to Highlighted successes, and explored why some collaborations failed to achieve anticipated outcomes – unintended outcomesachieve anticipated outcomes – unintended outcomes
Identified a number of learning points for future collaborative projects – Identified a number of learning points for future collaborative projects – client able to refine the model for new funding bidclient able to refine the model for new funding bid
Final Report completed in time for client to use as part of discussions Final Report completed in time for client to use as part of discussions with funder for future phases – powerful tool, evidence success and with funder for future phases – powerful tool, evidence success and demonstrate commitment to continual improvementdemonstrate commitment to continual improvement
Proactive approach to evaluationProactive approach to evaluation
Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace
Skills in the Workplace, launched in June 2005, was a Welsh Skills in the Workplace, launched in June 2005, was a Welsh Assembly Government initiative designed to Assembly Government initiative designed to “raise skill levels of “raise skill levels of employees and create an ethos of training within SMEs in North employees and create an ethos of training within SMEs in North Wales”Wales”..
The project attracted in excess of £4 million of European Social The project attracted in excess of £4 million of European Social Fund (ESF) support, and ran until July 2008.Fund (ESF) support, and ran until July 2008.
FE Colleges and Private Training providers offering ‘bite size’ FE Colleges and Private Training providers offering ‘bite size’ chunks of training to SMEschunks of training to SMEs
Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace
Comprehensive Formative Evaluation, run over 3 stages – Comprehensive Formative Evaluation, run over 3 stages – Baseline, Mid-point, FinalBaseline, Mid-point, Final
4 Thematic Reports – confusion in the marketplace, size of 4 Thematic Reports – confusion in the marketplace, size of company, sectoral analysis, modes of deliverycompany, sectoral analysis, modes of delivery
Research Methodology:Research Methodology: KI Interviews KI Interviews Employer SurveyEmployer Survey Company Case StudiesCompany Case Studies Desk Research/ MI AnalysisDesk Research/ MI Analysis Regular attendance at steering group meetings Regular attendance at steering group meetings Dissemination/Stakeholder eventsDissemination/Stakeholder events Comprehensive reporting Comprehensive reporting
Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace
Outcomes for the client:Outcomes for the client:
Formative evaluation model, measuring distance travelled and Formative evaluation model, measuring distance travelled and allowing partners to identify issues/opportunities at every allowing partners to identify issues/opportunities at every stage of delivery – integrated approachstage of delivery – integrated approach
Thematic reports offered in-depth analysis of key themes Thematic reports offered in-depth analysis of key themes which impacted upon or resulted from the projectwhich impacted upon or resulted from the project
Conclusions and Recommendations used to inform and Conclusions and Recommendations used to inform and evidence decisions for future business support modelsevidence decisions for future business support models
Evaluation was at the heart of dissemination efforts which Evaluation was at the heart of dissemination efforts which celebrated successes and reflected upon lessons learntcelebrated successes and reflected upon lessons learnt
Evaluation PrinciplesEvaluation Principles
In practice evaluation is seldom In practice evaluation is seldom “pure”“pure”
It reflects real life situationsIt reflects real life situations
It’s done for a mix of purposesIt’s done for a mix of purposes
Combines a number of types and design optionsCombines a number of types and design options
Measures at a number of levelsMeasures at a number of levels
Utilises a mixture of techniques (triangulation)Utilises a mixture of techniques (triangulation)
It allows judgements to be madeIt allows judgements to be made
Evaluation is not an exact scienceEvaluation is not an exact science
Why Evaluate?Why Evaluate?
TO LEARNTO LEARN
Demonstrate/plan/improve utilisation of resourcesDemonstrate/plan/improve utilisation of resources
Provide evidence to fundersProvide evidence to funders
Develop provisionDevelop provision
Inform policy developmentInform policy development
AccountabilityAccountability
Because we have to?Because we have to?
When do you evaluate?When do you evaluate?
Taken from: The Green Book: Appraisal and Evaluation in Central Government. Treasury Guidance
What do we really What do we really want to know?want to know?
Long-term impact or Long-term impact or short term outcomesshort term outcomes
Areas for Areas for improvement improvement
‘‘Next time’Next time’
Policy reviewPolicy review
Ex-Ante (feasibility, what if)Ex-Ante (feasibility, what if)
Formative (action research/ could it be better/different?)Formative (action research/ could it be better/different?)
Summative (what we did/achieved)Summative (what we did/achieved)
Process (how was it implemented)Process (how was it implemented)
Output (what was delivered)Output (what was delivered)
Outcome or Impact (what difference it made)Outcome or Impact (what difference it made)
Goal free (unintended consequences)Goal free (unintended consequences)
Economic Appraisal Economic Appraisal
(Builds on, covers more than monitoring)(Builds on, covers more than monitoring)
Evaluation OptionsEvaluation Options
Design OptionsDesign Options
Desk Research (what’s already known)Desk Research (what’s already known)
Data Reanalysis (what’s available already?)Data Reanalysis (what’s available already?)
Cross-sectional (cost-effective sampling)Cross-sectional (cost-effective sampling)
Comparative (between groups/contexts)Comparative (between groups/contexts)
Longitudinal (over time)Longitudinal (over time)
Some/ all of the aboveSome/ all of the above
Unit of AnalysisUnit of Analysis
Small/ larger areasSmall/ larger areas
Stakeholder groupStakeholder group
Individual/cohort (farmer or group of farmers, offenders)Individual/cohort (farmer or group of farmers, offenders)
Sector (farming, all farmers, prisons or prisoners)Sector (farming, all farmers, prisons or prisoners)
The economy (all economic outputs)The economy (all economic outputs)
Qualitative TechniquesQualitative Techniques
Semi-structured data collectionSemi-structured data collection
Focus groups, telephone and face-to-face interviewsFocus groups, telephone and face-to-face interviews
ObservationsObservations
WorkshopsWorkshops
DiariesDiaries
Data collection – topic guides vs. questionnairesData collection – topic guides vs. questionnaires
AnalysisAnalysis ThematicThematic Content AnalysisContent Analysis EthnographicEthnographic
Quantitative TechniquesQuantitative TechniquesStatistical analysisStatistical analysis
Surveys – CAPI, telephone, face-to-face, postal, Surveys – CAPI, telephone, face-to-face, postal, internet/emailinternet/email
A set of clear questions to poseA set of clear questions to pose Hard dataHard data
Census – fact basedCensus – fact based
Statistical returns/ MISStatistical returns/ MIS
Trends/statistical reanalysisTrends/statistical reanalysis
Enhance primary dataEnhance primary data
Data Collection (1)Data Collection (1)AdvantagesAdvantages
Face to face interviews:Face to face interviews: Good for open-ended issuesGood for open-ended issues Good (80%) response ratesGood (80%) response rates
Group interviews:Group interviews: Excellent for open-ended Excellent for open-ended
issuesissues May be developmental in May be developmental in
themselves (staff)themselves (staff)
Focus groups/community panels:Focus groups/community panels: Good for exploring issuesGood for exploring issues Possible to standardisePossible to standardise Possible to preserve anonymityPossible to preserve anonymity
DisadvantagesDisadvantages
Difficult to analyseDifficult to analyse Resource heavy (for researcher & Resource heavy (for researcher &
interviewee)interviewee)
Difficult to organise and analyseDifficult to organise and analyse Need trained interviewersNeed trained interviewers Useless for “competitors”Useless for “competitors”
Needs trained facilitatorsNeeds trained facilitators Difficult to analyse accuratelyDifficult to analyse accurately May be to balance the groupMay be to balance the group
Data Collection (2)Data Collection (2)
AdvantagesAdvantages
Telephone interviews:Telephone interviews: Good is the topics can be Good is the topics can be
defineddefined Quick (if you can get through)Quick (if you can get through)
Postal questionnaire:Postal questionnaire: Easy to administerEasy to administer Ensures privacyEnsures privacy Good for standardised dataGood for standardised data
DisadvantagesDisadvantages
Poor for qualitative datePoor for qualitative date Resistance to SUGOMResistance to SUGOM
Need to define questions very Need to define questions very tightlytightly
Low response rateLow response rate Unable to control who fills in the Unable to control who fills in the
formform
ReportingReporting
To whom?To whom?
For what purpose?For what purpose? Brief report to a meeting?Brief report to a meeting? Feedback to participants/clients/more widely?Feedback to participants/clients/more widely? Part of an on-going process?Part of an on-going process? Need for standardisation?Need for standardisation? Multiple audiencesMultiple audiences
SummarySummary Main reportMain report Extended analysisExtended analysis
Feedback to respondentsFeedback to respondents
EvaluationEvaluationDo’s & Don’tsDo’s & Don’ts
Do…Do…
Establish key research questions from the outset – what do you really Establish key research questions from the outset – what do you really want to know?want to know?
Collect good baseline dataCollect good baseline data
Clear specification will ensure better quality of tendersClear specification will ensure better quality of tenders
Consider the pro’s and con’s of internal/external evaluationConsider the pro’s and con’s of internal/external evaluation
How do you want the evaluation to work – is it to satisfy the funder, How do you want the evaluation to work – is it to satisfy the funder, or do you want it work as a project management tool or do you want it work as a project management tool
Make evaluation work for youMake evaluation work for you
Make sure you set aside adequate time and resources to implement Make sure you set aside adequate time and resources to implement and manage the evaluationand manage the evaluation
Involve stakeholders in the evaluationInvolve stakeholders in the evaluation
Disseminate and act on findingsDisseminate and act on findings
Don’t…Don’t…
Think about evaluation too late - unlikely to be effectiveThink about evaluation too late - unlikely to be effective
Send out ITT with ill-defined ideas and requirements – difficult for Send out ITT with ill-defined ideas and requirements – difficult for tenders to respond to effectively; difficult to compare tenderstenders to respond to effectively; difficult to compare tenders
Leave evaluators working in isolation – manage and learn from Leave evaluators working in isolation – manage and learn from the processthe process
Cast the net too wide when commissioning evaluators – wastes Cast the net too wide when commissioning evaluators – wastes time and resources for all time and resources for all
Lose sight of why you are evaluatingLose sight of why you are evaluating
Treat evaluation as a bolt-onTreat evaluation as a bolt-on
Ignore findings – act on themIgnore findings – act on them
Thanks for your time Thanks for your time
Happy to receive comments or queriesHappy to receive comments or queries
Mark BeynonMark BeynonCRG ResearchCRG Research
25 Cathedral Road25 Cathedral RoadCardiffCardiff
CF11 9TZCF11 9TZ
029 2022 3218029 2022 [email protected]@crgresearch.co.uk