Theory Driven Evaluation: tracing links between

assumptions and effects

Sixth European Conference on Evaluation of Cohesion PolicyWarsaw, 30 November – 1 December 2009

Karol Olejniczak, k.olejniczak@uw.edu.plEUROREG – University of Warsaw: www.euroreg.uw.edu.pl

Theory-Driven Evaluation in a nutshell• It relates evaluation research with the

scholarly socio-economic theories• It treats programme as optimistic assumption

about causal relations: If we DO…. than we GET… and than…

• Programme is a set of theories…– Underlying Theories (knowledge, experience, influences)

– Theory of Change (assumption about strategic change)

– Theory of Implementation (the way every-day work is organized)

• ...that works in a certain context & circumstances

• TDE is and approach, it is method-neutral– Logic models– 5 stages procedure

5 stages procedure

Theory Driven Evaluation in PracticeCase Study of Ex post Evaluation Neighbourhood Program INTERREG/TACIS CBC Poland-Belarus-Ukraine 2004-06

Programme• Border regions from Poland, Belarus, Ukraine• Two programmes – two pots of money: INTERREG 37,8 mln EUR

vs TACIS 8 mln EUR• Beneficiaries – local communities, local services, NGO• 173 major projects + 300 micro-projects

Contracting Authority:

• Polish Ministry of Regional Development, Territorial Cooperation Unit

Evaluator: • EGO – Evaluation for

Government Organizations s.c.

Contract: • 5 monhts (July-Nov 09)

Stage 1: Understanding the context

Questions:

• What theories tell us about cross-border cooperation?

• What type of border it is?

• What was the scale of intervention?

Answers:

• CBC depends on permeability of the border & partners complementarities

• Highly peripheral, underdeveloped area, border as barrier

• Minor financial impulse, could be visible only on a local scale

Methods:• Literature review, review of earlier empirical research• Analysis of general statistics• Review of socio-political situation

• Interviews with the Directors of the Departments

• Secondary data– Effectiveness indicators from Task-oriented budgets– Staff turnover– Other

Stage 2: Defining Theory of Change & tracing its Underlying TheoriesQuestions:• What strategic objectives

have been set?

• What were the assumptions and inspirations for these decisions?

Answers:• Dual objective: IF we act on 5

thematic fields THAN quality of life AND socio-economic integration will be improved in a border area

• Reasonable fields of intervention but no measures for border permeability

• Stakeholders choices, intuition, earlier experiences of Poland-Germany CBC ProgrammeMethods:

• Desk research (programme documents)• Interviews with key programme stakeholders involved in

the programme design

• Interviews with the Directors of the Departments

• Secondary data– Effectiveness indicators from Task-oriented

budgets– Staff turnover– Other

Stage 3: Reconstructing detailed Theory of Change and its indicatorsQuestions:• How should we define the

change in terms of indicators & assessment criteria?

• How programme designers defined these changes in terms of programme & projects indicators?

Answers:• Effects both planned and side-

effects have to be measured on 2 scales: quality of life & integration

• They have to be traced in 3 dimensions: thematic (projects topics), relational (partnerships) and territorial (local communities)

• Analysis & assessment criteria base on logic models, they differ in details but scale stays the same

Methods:• Logic models for each thematic group and partnerships• Assessment system – two scales: quality of life vs

integration• Review of the programme indicators

Stage 4: Tracing real changes – outputs & effects

Questions:• What was the funds spatial

& thematic distribution?• What was the number &

structure of partnerships?• What were the effects of

thematic group of projects?• What were the effects of

partnerships?• What were the impact on

local communities?

Answers:1. High disproportion between 3

sides of the border2. Focus on improving quality of

life3. Local and close-to-border

effects4. The main integration effect

was brought by soft-projects, it was institutional, limited integration of local societies

5. Balanced effects (quality + integration) brought by tourism & border security projects

Methods:• Local visits of all project sites

(different observation tools applied, depending project’s topic)

• Survey of project beneficiaries, survey of partnerships• Social survey of twin communities• Interview with local stakeholders, review of local press

Stage 5: Explaining the obtained effects

Questions:1. Why disproportion?2. Why focus on quality of

life?3. Why close-to-border and

local effects?4. Why institutional

integration?5. Why tourism & border

security projects had best effects?

Answers:1. Unbalanced money & procedures;

higher experience of Polish teams2. High peripheriality & local needs;

unclear demarcation line3. Border as a sealed barrier, small

funds spread spatially4. Selection criteria, limited trust –

focus on smaller projects, expert-type projects, micro-projects as top-down not bottom up initiatives

5. Tourism easy to combine with next initiatives (multiplier effects), security required official cooperation agreements

Methods:• Brainstorming with experts• Second review of qualitative & quantitative data again• Interviews with programme managers• Survey of unsuccessful and potential applicants

• Interviews with the Directors of the Departments

• Secondary data– Effectiveness indicators from Task-oriented

budgets– Staff turnover– Other

Summing up the case study

Main message:• Close to border and local effects. • In a given context every joint project was a

success

How TDE helped us?• Dealing with complexity - packing &

unpacking issues• Focusing exploration on the right level • Making the fair judgement – understanding

contextual limitations• Writing the concise report - clear narrative

TDE for cohesion programmes

Advantages

• Articulates rationality of the programme

• Provides clear conceptual foundation for the study

• Focuses on effects and treats implementation issues only as one of the explaining factors

• Relates to scholarly theories and give bigger picture

• Allows to discuss causal relations

Challenges

• Using too rigid model can lead to tunnel vision and omitting side-effects

• There is a trade-off between level of details and clarity of the models

• Too much sophisticated theoretical considerations can alienate stakeholders and turn evaluation into scholar research

Bibliography

• Chen, H.T. (2004) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousands Oaks: Sage Publications, Inc.

• Donaldson, S.I. (2007) Program Theory-Driven Evaluation Science: Strategies and Applications. New York: Lawrence Erlbaum.

• Knowlton, L.W. & Phillips, C.C. (2008) The Logic Model Guidebook: Better Strategies for Great Results. Los Angeles: Sage Publications, Inc.

• Leeuw, F.L. (2003) Reconstructing Program Theories: Methods Avaliable and Problems to be Solved. American Journal of Evaluation, 24(1), pp.5-20.

• Patton, M.Q. (2008) Utilization-focused evaluation. 4th edition. Los Angeles, London: Sage Publications.

• Pawson, R. (2009) "Introduction to Realist Evaluation and Realist Synthesis", lecture on The Academy of Evaluation, EUROREG – University of Warsaw, Warsaw, 7.02.2009

• Weiss, C.H. (1997) How can theory-based evaluation make greater headway? Evaluation Review, 21(4), pp.501-524.

• Weiss, C.H. (2004) On Theory-Based Evaluation: Winning Friends and Influencing People. Evaluation Exchange, IX(4), pp.2-3.

Contact details

Karol Olejniczak, PhDEUROREG – University of Warsawwww.euroreg.uw.edu.pl • E-mail: k.olejniczak@uw.edu.pl • Phone: +48 22 826 16 54• Mobile: +48 696 41 22 82