View
647
Download
0
Category
Preview:
Citation preview
EVALUATION OF SME AND ENTREPRENEURSHIP PROGRAMMESJonathan Potter and Stuart Thompson
Trento13th November 2015
Structure of the presentation
1. Role and importance of evaluation2. Approaches to evaluation – measures and
techniques3. Evaluation examples4. Planning and organising evaluation5. Some conclusions
Part 1
Role and importance of evaluation
OECD Istanbul Ministerial Declaration
Recognised“The need to develop a strong evaluation culture in ministries and agencies responsible for SME policies and programmes” “Evaluation provides a means of ensuring that SME programmes remain cost-effective and adapt to changing conditions in a dynamic world”
What do we mean by evaluation?
Papaconstantinou and Polt (1997): “Evaluation refers to a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effectiveness of an activity in terms of its objectives” Purpose is to provide decision makers with the best information to answer crucial questions, e.g.• Does my programme address the key
issues? • What is working and not working? • What can I do better next time?
Why evaluate?
• Learning from experience• To establish the impact of policies and
programmes against their objectives• To make informed decisions about the
allocation of funds across a portfolio• To show the taxpayer whether the
programme is a cost-effective use of public funds
• To stimulate informed debate• To achieve continued improvements in the
design and administration of programmes
Some evaluation questions
• Is the activity still relevant? • Is it achieving its objectives? • Is it cost effective and proportional to
what it seeks to achieve? • Should the activity be continued or
terminated? • If continued, how can it be improved?• Can it be used in different contexts?• What issues will occur when scaling up a
pilot project?
Part 2
Approaches to evaluation – measures and techniques
Inputs
Resources required for the programme (finance, people etc)
Often defined in advance, needs to be sufficient
↓
Activities
Actual actions taken (training courses, advice services etc)
Determined from technical considerations and experience
↓
Outputs
Immediate measure of activities (number of trainees, people advised)
Determined from previous experience and evaluations
↓
Outcomes
Change in behaviour of target (number of start-ups etc)
Requires detailed analysis and research
↓
Impacts
Overall effect (noting what would have happened anyway etc)
Often only calculable well after the programme is complete
Measures at different levels
Measure Definition Typical QuestionsRelevance The extent to which the activity is
suited to the priorities and policies of the target group, recipient and government.
To what extent are the objectives of the programme still valid?Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?
Effectiveness The extent to which the intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance.
To what extent were the objectives achieved / are likely to be achieved?What were the major factors influencing the achievement or non-achievement of the objectives?
Efficiency The outputs in relation to the inputs. It signifies that the intervention uses the least costly resources possible in order to achieve the desired results.
Were activities cost-efficient?Were objectives achieved on time?Was the programme or project implemented in the most efficient way compared to alternatives?
Impact
The positive and negative changes produced by an intervention, directly or indirectly, intended or unintended on the local social, economic, environmental and other development indicators.
What has happened as a result of the programme or project?What real difference has the activity made to the beneficiaries?How many people have been affected?
Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.
To what extent did the benefits of a programme or project continue after funding ceased?What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?
Typical evaluation measures
Monitoring Step I Take up of program Step II Recipients’ opinions Step III Recipients’ views of the differences
made by the program Evaluation Step IV Comparison of the performance of
‘assisted’ with ‘typical’ ventures Step V Comparison with ‘matched’ ventures Step VI Taking account of selection bias
Approach - six steps
The ideal approach – Randomised control group
Example - GATE
• Open to all, but targeted at the unemployed
• Applicants randomly given assistance or not
• No barriers to using other sources of advice
• Result showed robust evidence of an impact
Impaq International (2009), Growing America Through Entrepreneurship: Final Evaluation of Project GATE
Regression model
N
NN N
N
N N
N
NN
N
N
NN
N N
N
NN
N
B
B
BB B
BBB
B
BB
B
BB
B
B
B
B
Out
put v
aria
ble
(res
ult o
f a v
arie
ty o
f fa
ctor
s af
fect
ing
firm
cha
ract
eris
tics)
Firm characteristics (control variables and value of policy support)
Beneficiaries
Non-beneficiaries
Matching (propensity scoring)
Differences in differences
Regression discontinuity
Part 3
Evaluation examples
Example – Prince’s Trust Youth Business Initiative
Greene, Francis J (2005), Evaluating Youth Entrepreneurship: the Case of the Prince’s Trust
• The Prince’s Trust is one of the longest running programmes to support youth entrepreneurship in the United Kingdom. It uses a mix of soft (mentoring) and hard support (loans and grants) and has maintained the same approach over a long period.
• The programme’s specific aims are:• Supporting individuals who would be unlikely
to start up an enterprise• Deepening the human capital of participants• Improving survival rates
Prince’s Trust - methodology
• Survey of recipients• Survey of non-recipients• Survey of non-survivors• Matched survey with sample
selection• 3600 respondents• Survival rates assessed over 3 years
Prince’s Trust - results
• Only mentoring appears to have a positive effect
• This is at variance with simpler evaluations which only looked at the views of participants, who were very positive
• No evidence that entrepreneurship support assisted employability
Example – Start-up grants to unemployed immigrants in Germany
• Germany has had long term policies of providing support and “welfare bridges” to the unemployed i.e. continued subsidies even when they are self-employed. Exact levels of benefits have varied over time.
• Immigrants have different characteristics from the general population and may respond to policies differently.
• There have been a number of positive evaluations of the scheme as a whole, but these do not mean that it works in the same way for all groups of the unemployed, for example immigrants.Robinson, Eric (2011), Start-Up Grants to Unemployed Immigrants in
Germany: A Means to Reach Employment Parity?
Methodology
• Use of long term socio-economic database, the German Socio Economic Panel – including more than 140,000 individuals tracked between 1983 and 1993.
• Regression techniques, controlling for sex, age, marital status, education, work experience, sector, type of self-employment
Results
• Unlike natives, foreigners are more likely to lapse back into unemployment when start-up benefits are high. This difference is statistically significant.
• Consideration of skill levels suggests that this is because high benefits give an incentive for ill-prepared individuals to start up business.
Part 4
Planning and managing evaluations
Evaluation and the policy cycle
ProgrammingWhat is the objective?
IdentificationWhat are the barriers?
FormulationHow can we address it?
ResourcingCan we mobilise resources?
ImplementationIs it on track?
Ex Post EvaluationDid it produce the results?
Ex Ante EvaluationOngoing/Interim
Evaluation
Meta evaluationsBenchmarking
Terminal Evaluation
• A timetable for evaluations• Responsibility for commissioning or
performing evaluations• Quality assurance• Mobilisation of resources
Evaluation Framework - management
• Identifying stakeholder needs• Deciding what to evaluate• Key Performance Indicators• Timely and accurate information• Qualitative information• Information regarding assumptions• Baselines• Counterfactual
Evaluation framework - information
Getting the information?Cost
Complexity
Administrativereports
Managementreports
Availablestatistics
Adapted availablestatistics
Interviews withbeneficiaries
Specialisedsurveys
Dealing with the results
• Is there logic in the report?• Is it peer-reviewed? Quality assured?• Have things changed?• What needs to be public?• What is politically necessary?
Part 5
Conclusions
Conclusions
• It is important to evaluate consistently and robustly across programmes
• Effective and efficient evaluation requires planning from the beginning of implementation
• Evaluation often involves deciding on proportionality – the amount of effort involved in gathering data against the usefulness of the result
• Managing authorities need to develop capacity in commissioning evaluations rather than in actually doing them
• Full impact evaluations can be difficult
Questions
Recommended