On Track: The evaluation of influence in system wide change

Content

• Introduction to ‘On Track’

• The evaluation

• The findings

• Theory of change

• Organisational & community readiness

• Potential next steps

• Reflections

Influence

“The capacity or power of persons or things to

produce effects on others by intangible or

indirect means”.

Kirkhart, K. (2000) Reconceptualising evaluation use: An integrated theory of influence. New Directions for Evaluation. No 88 (pp 5-24)

Images of the future - 5 archetypes

1. Evolution and progress - more technology.

2. Collapse - we have reached our limits.

3. Gaia – the world is a garden or a forest.

4. Globalism – breaking down borders.

5. Back to the future – returning to simpler times.

The eco-cycle of change

Source: Mark Cabaj (2013) The eco-cycle of change. The Collective Impact Institute.

Rigidity trap

The MH&A sector over time

Change pressures in the present

Source: Platform Trust & Te Pou o Te Whakaaro Nui (2015, p30) On Track. Knowing where we are going.

On Track – 7 priority action areas

The evaluation

Purpose of the evaluation

• ‘On Track’ was published in April 2015.

• This was a developmental evaluation that was designed to give the project team a sense of whether or not we were ‘on track’.

• The information was also intended to help inform what we need to do in order to continue to influence change.

Methodology

• Literature review of relevant behaviour change theories.

• A review of On Track documentation.

• Key informant interviews (Te Pou & Platform Trust).

• Survey of key informants from 13 District Health Boards.

• Follow-up telephone interviews.

• Workshop with staff from a selected DHB ‘success case’.

Districts engaged through On Track workshops

On Track workshops:Estimated numbers of participants from May 2015 - Sept 2016.

The findings

Charting significant positive change with On Track being identified as an important cause (n=9)

Examples of significant change – On TrackDistrict Health Board responses NGO provider responses

• Working in an inter-sectoral way.• More emphasis on delivering services

closer to where people live.• Increased consumer & family

participation through a consumer group.

• Shared leadership model (DHB & NGO).• A work plan that is influenced by On

Track and Rising to the Challenge.• Shared training opportunities across all

mental health and addiction services.• New co-designed and/or co-ordinated

services.

• Increased emphasis on people determining their own health pathways.

• Increased inter-sectoral approaches.• Innovative models of service delivery.• Connecting with other organisations to

improve service integration and to develop peer support, including suitable staff training.

WHAT HELPS & HINDERS POSITIVE CHANGEWHAT TO FOCUS ON?

Enablers Barriers

Those positive factors that appear to collectively enable innovation and the implementation of better and more integrated health & social policies and services.

Those negative factors that halt progressor undermine and slow change efforts.

• Predisposing factors – those that create the conditions for change.

• Perpetuating factors – those that amplify and normalise a change state.

WHAT HELPS & HINDERS POSITIVE CHANGE

Enablers Barriers

• Policy responses to systemic failures.• Policies that set a new national direction.• Workforce capability-building.• External change agencies that seek to

influence system change.• Strategic leadership at the national level.• Internal change agents – local advocates

for systems and service change.• Disposition to collaboration/community

engagement at the district level.

• Systemic shock (eg, the impact of sentinel events).

• Systemic stress – no slack in the system, scare resources, workload issues and endemic lack of optimism. The workforce operating in survival mode.

• Challenges associated with rural areas –eg, physical isolation and stretched infrastructure.

WHAT HELPS & HINDERS POSITIVE CHANGE (contd?Enablers Barriers

• Strategic leadership. Holding a vision and having a mandate for change at the local or district level.

• Lack of executive team buy-in.

• On Track as an intervention promoting a common agenda for change (ie, actionable).

• Having local On Track champions.• Change leadership & adaptive management,

including ‘learning by doing’ and the ability and willingness to change tack.

• Network formation & teamwork.

• Tendency to independence vs, partnering.• Underdeveloped feedback systems. • Limited mechanisms for deliberative,

responsive and progressive change. • Lack of persistent external support.

• A sense of organisational and community self-efficacy.

• Integrative technology serving multiple purposes, including rapid system feedback.

• Losing champions - people familiar with and committed to sector development (eg, shared vision/common agenda). NB: This may be a function of a negative milieu or low self-efficacy.

Why & how On Track has been influential

Qualities of On Track as an innovation

• Enacts government policy.

• Compatibility – expresses a shared vision/common agenda.

• Provides an unambiguous advantage over the status quo.

• Adaptable to organisational & community needs – not prescriptive.

• Easy to use – certain components can be taken up opportunistically, reducing complexity and risk.

Theory of change

DR

AFT

Th

eo

ry o

f C

han

ge

.

.

.

DRAFT model

.

Leadership

Reliant on the personal attributes of leaders and

managers understanding the dynamic and multi-

dimensional nature of the MH&A system.

Tactical-level leadershipManagers engaging critics,

enabling innovators and sharing leadership (ie,

distributed leadership).

Strategic leadershipOwning and articulating a

vision for change.

Adaptive managementManagers driving learning and change, based on the evidence and a process of

systematic inquiry.

.A

gen

da

sett

ing

Behaviour Change Frameworks

Source: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-8-46

Fostering a collaborative culture

Checklist of behaviours for collaborative leaders:

• Encourage the process of goal, role and task clarification.

• Promote open communication in order to achieve team goals.

• Develop collaborative team norms – operate with fewer rules but set high standards.

• Take personal responsibility for the team’s success.

• Employ energising strategies.Source: Weiss, et al. (2014) The Interprofessional Health Care Team.

Organisational & community readiness for change

WHAT HELPS & HINDERS POSITIVE CHANGEOrganisational and community readiness

The importance of organisational and community readiness with regard to the combination of enablers and barriers that are in play.

Note: There is considerable variation between New Zealand District Health Board areas in their overall readiness for change.

Organisational and community readiness model

DHBs could consider their ‘readiness’ from the following three angles:

1. Contextual readiness - their capacity for change.

2. The level of engagement with the vision outlined in On Track.

3. Subsequent developments in the direction of On Track (either caused by On Trackand/or by other contributing factors).

Source: http://peersforprogress.org/resource-guide/assessing-community-readiness-for-innovative-strategies/

Potential next steps

Potential next steps

• Support districts to use theoretical frameworks for diagnosing system-related problems and appropriate change strategies.

• Support districts to implement a baseline and benchmarking process.

• Undertake outcomes harvesting across designated districts.

• Support local change agents to undertake evaluative monitoring activity (both qualitative and quantitative).

• Compile stories of success and significant change.

Reflections

Some reflections

• Evaluating ‘influence’ is more difficult than evaluating ‘a thing’.

• Thinking systemically– boundaries, perspectives & interrelationships.

• The interrelational aspect of the MH&A system has, to some extent, evaded easy capture.

• Self-organising complex systems -paying attention to emerging patterns and then responding to them.

“We are now in a transition from an

object-oriented to a system-oriented

culture. Here change emanates, not

from things, but from the way things

are done.”

Jack Burnham – Systems Aesthetics (1968)