Supporting continuous improvement in the replication process Getting to Grips with Replication Seminar 3: Monitoring, evaluation & continuous improvement

Supporting continuous improvement in the replication

processGetting to Grips with Replication

Seminar 3: Monitoring, evaluation & continuous improvement

21st March 2013

Kerstin Junge, Tavistock Institute of Human Relations

Introduction• Why is it useful to think of replication in

continuous improvement terms? • How can we use monitoring and evaluation as

tools for continuous improvement in the replication process?

• How do these tools affect impact of the replication process?

• Conclusions

Why is it useful to think of replication in continuous improvement terms?

Continuous improvement

Replication of social interventions

Incremental or breakthrough process

Incremental process

Phases of small scale testing, measuring, acting (wider roll-out or ongoing testing) in commonly used PDSA cycle: Plan identify an opportunity and plan for

change Do: implement the change on a small scale Study: use data to analyse the results of the

change and determine whether it made a difference

Act: if the change was successful, implement t on a wider scale and continuously assess your results. If the change did not work, begin the cycle again.

Takes place in three phases: Creating or identifying an innovation for

replication (including evaluation to show it works)

Deciding to adopt an innovation for replication (including identifying funding)

Implementing, replicating, and measuring success of the intervention (in waves)

Relies on monitoring data to inform change generated from exercises

Uses both monitoring and evaluation data to inform adoption and success of replication process

How can we use monitoring and evaluation as tools for continuous improvement in the

replication process?

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition

Replication Phase 1: Identifying innovations for replication and deciding to adoptReplication Phase 1: Identifying innovations for replication and deciding to adopt

Requirement to engage with existing impact evidence as

integral part of programme design









“The application stage forced me to ask the… team: why are we doing it in this way? How do we know it works? It was challenging to ask the organisation. (…)

These are hard questions. Realising Ambition clarified the thinking in the

organisation.”

“The application stage forced me to ask the… team: why are we doing it in this way? How do we know it works? It was challenging to ask the organisation. (…)

These are hard questions. Realising Ambition clarified the thinking in the

organisation.”

“The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention). They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather than external people so we are using teachers.”

“The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention). They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather than external people so we are using teachers.”

Application process challenged us, (…) it asked for some quite robust

information and we had to organise ourselves to get that. [It] challenged

us to smarten up back end, e.g. pulling the logic model together;

we knew it and played around with doing that and had not expressed

that in a document. (…) we actually said even if we don’t get that, [the

application process] actually helped us organise ourselves (…).”


information and we had to organise ourselves to get that. [It] challenged

us to smarten up back end, e.g. pulling the logic model together;

we knew it and played around with doing that and had not expressed

that in a document. (…) we actually said even if we don’t get that, [the

application process] actually helped us organise ourselves (…).”







“The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention).

They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather

than external people so we are using teachers.”




“The application stage forced me to ask the… team: why are we

doing it in this way? How do we know it works? It was challenging to ask the organisation. (…) These

are hard questions. Realising Ambition clarified the thinking in

the organisation.”






information and we had to organise ourselves to get that. [It] challenged us

to smarten up back end, e.g. pulling the logic model together; we knew it

and played around with doing that and had not expressed that in a document.

(…) we actually said even if we don’t get that, [the application process]

actually helped us organise ourselves (…).”







Creating preconditions for impact through: better and explicit knowledge of the intervention and effective delivery, clearer understanding of causal pathways, preparing the organisation for replication


Embeds evidence based

and reflective practice in

programme community


Process evaluationSense-making and capturing of lessons learnt to support future replication investments

Analysing programme architecture, programme design and application process


Analysing programme architecture, programme design and application process


































Embeds evidence based

and reflective practice in

programme community


Replication phase 2: first wave implementationReplication phase 2: first wave implementation

Monitoring systemMonitoring system Keeping programme on track, understanding challenges, improving deliveryKeeping programme on track, understanding challenges, improving delivery



Preparation for (new) impact evaluations


Monitoring systemMonitoring system

“One of the most significant things was (…)

looking at how we constructed the logic

model. (…) Now it's much easier to be specific.”




Webinars “(…) spark thinking which can feed into plans and

keep them on track.”

Webinars “(…) spark thinking which can feed into plans and

keep them on track.”

“Some of the work that is being on theory of change; understanding

replication and fidelity. What [intervention owners] do is set up the programme, train the staff and give us assistance. But they don’t

engage staff in discussing in understanding programme fidelity and replication. “

“Some of the work that is being on theory of change; understanding

replication and fidelity. What [intervention owners] do is set up the programme, train the staff and give us assistance. But they don’t

engage staff in discussing in understanding programme fidelity and replication. “

Keeping programme on track, understanding challenges, improving deliveryKeeping programme on track, understanding challenges, improving delivery






Probability of achieving desired impact increases through: greater specificity of intervention and expected outcomes as well as improved understanding the importance of fidelity in

delivery.


delivery.







Webinars “(…) spark thinking which can feed

into plans and keep them on track.”



“Some of the work that is being on theory of change;

understanding replication and fidelity. What [intervention

owners] do is set up the programme, train the staff and

give us assistance. But they don’t engage staff in discussing in understanding programme fidelity and replication. “






Offers data, learning and

knowledge to guide and

improve replication

delivery




Identification and (early) definition of replication models; overview of types and benefits of impact evaluation and other support ; replication progress indicators


Identification and (early) definition of replication models; overview of types and benefits of impact evaluation and other support ; replication progress indicators





delivery.


delivery.




















Offers data, learning and

knowledge to guide and

improve replication

delivery

Effects on impact

• Creating preconditions for impact: – Better and explicit knowledge of the intervention and

effective delivery– Clearer understanding of causal pathways– Preparing the organisation for replication

• Greater probability of achieving desired impact:– Greater specificity of intervention and expected outcomes– Improved understanding the importance of fidelity in

delivery– Data, knowledge and learning to improve replication

delivery

Conclusions• Understanding replication as a continuous improvement process

offers a ‘practical’ framework to guide activities, as well as tools • Monitoring and evaluation tools should be an integral part of the

replication process from the start, both at project level and at programme / policy level– Creates a structure to support constructive engagement with

evidence, evidence based practice, learning from experience – And through this supports learning and continuous improvement

• It means understanding monitoring and evaluation at least as much about learning as about ‘auditing’

• Possibility of not only ‘single loop learning’ but also ‘double loop learning’ (Argyris and Schön)? – Not just continuous improvement of implementation (revisiting

‘action strategies’)– But also revisiting values [mental models, beliefs, intentions]

underpinning replication actions and strategies (‘governing variables’)

Documents

Supporting continuous improvement in the replication process Getting to Grips with Replication Seminar 3: Monitoring, evaluation & continuous improvement