PhD private defence: Realist evaluation of a capacity building programme for health managers in Tumkur, India

How do capacity building programmes work in local health systems?

A realist evaluation of a local health system strengthening intervention in Tumkur, India

Prashanth Nuggehalli Srinivas

Private defense UCLFebruary 20, 2015

Promoter: Jean MacqCo-promoter: Bart Criel

Outline

• Part 1: The big picture – Strengthening health systems in India

• Part 2: Local health systems and organisational change

• Part 3: Study setting and intervention

• Part 4: Methodology and study design

• Part 5: Analysis and results

• Part 6: Discussion, relevance and lessons learned

2

Background

Methods

Results & Discussion

The big picture

Strengthening health systems in India

3

Part 1

Indian health system is pluralistic & evolving

4Van Damme et al 2010

Part 1

Background

Regional improvements, but disparities remain

5

“Accelerated progress to reduce mortality during the neonatal period and at ages 1–59 months is needed in most Indian districts.”

- Usha Ram et. al. 2013

Part 1

Background

Regional/sub-regional (district-taluka) disparitiesRole of (poor) management?

For example, in 2006Immunisation coverage – 91% in Kodagu district and 70% in Raichur.

114 “backward” talukas, nearly half in “forward” districts

“Systemic failure” as a cause? (George, 2007&2009), Sen (2006)

6

Part 1

Background

Human resources for health

• Good health workforce –Available & competent

• Improved organisationaloutcomes through ‘good HRM’– Lifelong learning and supportive

(yet firm) supervision

– Competent and responsive managers who are able to manageresources and plan health care services

Part 1

Background

The National (Rural) Health Mission

8

Part 1

Background

• Poor planning and management contribute to disparate health outcomes in Indian districts

• Structural reforms such as the NRHM need well-performing district & sub-district local health systems

Prashanth N S (2013) BMJ Rapid Responsehttp://www.bmj.com/content/347/bmj.f5621/rr/662992 9

Understanding local health system performance

10

Local health systems

• More than a sum of the services; acquire a specific local character in view of their internal characteristics and the context

• Interface between top-down policies and bottom-up demands

• Locus for conceptualisingorganisational change through building capacity and improving performance

Van Olmen et. al. 2012

11

Part 2

Background

Capacity and performance

• Multi-dimensional nature of capacity & performance – individual, organisational, environmental

• Capacity & capacity building closely related to performance, but may not automatically improve performance – various dynamic interactions between internal and external factors influence performance 12

Part 2

Background

Brown, L., LaFond, A., & Macintyre, K. (2001). Measuring capacity building. Chapel Hill: MEASURE Evaluation.

Socio-culturalEffortTimeCulture-oriented change & “new way of doing things” (shifting norms, powers, values)

13

TechnicalProgrammaticQuick(er)TangibleTask-oriented nature of change (changing procedures and activities)

Potter & Brough 2004

Organisational changePart 2

Capacity-building as an HRM intervention

• Implemented with the objective of knowledge or skills transfer through training programmes

• Frequent calls for greater capacity-building in literature and some studies on effectiveness, but:– How do these programmes work at the “systems” level?

– Under what circumstances do these lead to behaviouralchange and improved performance of the organisations?

– What are the contextual elements that promote (or hinder) such change?

14

Part 2

Background

Rationale for the study

• The literature gap – Review of 28 European Commission-funded projects shows need for systemic capacity building & research; and scanty literature on how it works (Potter & Borough 2005)

• The evidence gap – poor evidence for structuring capacity-building interventions (Rowe et al 2005)

• The methodological gap – evaluation of complex HRM interventions

• Timeliness & relevance – National Rural Health Mission

15

Part 2

Background

Asking the how question in healthcare evaluations in India

• Review of health programmeevaluation in India

• Little inter-disciplinarity

• Heavy tilt towards “did programme work” and comparing coverage and effectiveness

• Two case studies – maternity benefit scheme & health insurance for people below poverty line Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating

Healthcare Interventions: Answering the “How” Question. Indian Anthropologist, 43(1), 35–50.

16

Part 2

Study setting and intervention

17

18

Public health services organisation in India

Part 3

Background

19

Part 3

Background

Delphi study by IPH, Bangalore on poor performance of district health services (2007)

Study setting

20

Part 3

Background

21

22

Mentoring

Contact classes & Assignment

Health managers (medical & non-medical) at District level – DHO, DS, Programme officers, DPM, nursing managers and senior admin staff

Health team at taluka level – THO, AMO, BPM, AAO

PHC Medical Officers, PRI members

2-3 days per month, residential contact classes

At least 5 mentoring days

1 assignment/month

Methodology and study design

23

Methodological considerations

• Programme theory and assumptions were not explicitly formulated

• From effectiveness to mechanisms of change in organisations

• Mixed methods study

24

Part 4

Methods

Scope for realist evaluation

Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating Healthcare Interventions: Answering the “How” Question. Indian Anthropologist, 43(1), 35–50.

25

Part 4

Methods

Realist approach

Mechanism: what is it about an intervention which may lead it to have a particular outcome in a given context?

Context: what conditions are needed for an intervention to trigger mechanisms to produce particular outcomes patterns?

Outcomes pattern: what are the practical effects produced by causal mechanisms being triggered in a given context?

Pawson & Tilley, 199726

Part 4

Methods

The realist cycle

27

Part 4

Methods

Three cycles

• Eliciting the PT

• Macro/meso level contextual conditions

• Contrasting cases within Tumkur

28

Part 4

Methods

Data collection

• Field notes of observations during classroom teaching, mentoring visits, district and talukareview meetings and supervision visits

• Interview with participants (7+7+8), supervisors (2), state-level bureaucrats (2) and implementers (2) in three episodes: early intervention, mid and post

• Secondary data: annual plans, district-planning guidelines from state and central government, programme documents of the NRHM

29

Part 4

Methods

Survey

– Attitude towards decentralised planning and training programmes

– Organisational commitment (Mayer & Allen)

– Self-efficacy (Bandura)

– Supervision (Oldham & Cummings as adapted from Michigan OA package)

– Respondents (Tumkur and a comparator district): 65+27

30

Part 4

Methods

31

Part 4

Eliciting the programme theory (PT)

• Described the process of refining PT– Understanding the

intervention (IPT)– Review of literature to

identify mechanisms reported

– Identify relevant contextual factors

– Refine PT– Formulate change scenarios

(C-M-O)

32

Part 4

Methods

KeyIPTassumption

Supportingtheory

Keycontextual

factor

Plausiblemechanism

identifiablefromIPTand

theory

Outcomeofinterest

Contactclasses’work

throughimprovingknowledgeand/orskills,whichareeventuallyapplied.Thisresultsinimprovedperformance

Outcomesoftrainingprogrammes

accruethroughfourhierarchicallevels:reaction(totrainingprogramme),learning,behaviourandimpact(KirkpatrickandKirkpatrick

1998)

Teamdynamics(natureof

teamandrelationships)affectstheindividualwithintentionforpositivechange

Motivationoftheparticipant

towardspositiveorganisationalchange-a“can-do”attitudeintheIPT

Intentiontomakepositive

changes

Context-mechanism-outcome 33

KeyIPTassumption

Supportingtheory

Keycontextualfactor

Plausiblemechanism

identifiablefromIPT

andtheory

Outcomeofinterest

Mentoring

participants

atworkplacefacilitates

applicationof

knowledge

andskills

Workplace

environmentin

healthcareorganisationshas

beenidentifiedasanimportant

elementthat

explainsapplicationoflearningfrom

trainingprogrammesin

somesettings,

whilenotinothers(Clarke

2005).

Natureof

supervisionand

district’sopennessto

“allow”change

Natureof

commitment

toorganisation

Identify/seek

opportunities

tomakepositive

changeintheorganisation’s

performance

Decentralised

actionplansanddecision-making

atdistrictand

lowerlevels.Stateandhigher

levels’opennesstochange

proposals

Self-efficacy Improved

annualactionplans–Better

situation

analysis,problem

identification,allocation

and

utilisationofresources

34Context-mechanism-outcome

KeyIPTassumption

Supportingtheory

Keycontextualfactor

Plausiblemechanism

identifiablefromIPTandtheory

Outcomeofinterest

Acapacitatedhealthmanagercanbecomeanagentofpositiveorganisationalchange

Highcommitmentmanagementliteratureshowsthepotentialforchangebycommittedstaffinsettingswhere

resourcescouldbemobilised(Marchal,Dedzo,andKegels2010a).

Changeproposalsbydistrictsareinlinewithstate(orcentral)visionaswellasaddresslocalneeds.(Allocationandstrategicalignmentwith

externalenvironmentperSicotteetal.’sconceptualframework)(Sicotteetal.1998)

Claimingandutilisingdecisionspaces;organisationalcommitmentandself-efficacyinnegotiatingwithsuperiorsand

communityleaders

Talukaanddistrictsplanimproves.Theyidentifymoreneeds,mobilisemoreresourcesfromstateandutiliseitbetter

(Efficiency–bothallocativeandtechnical–improves)

35Context-mechanism-outcome

Analysis and results

36

Elicited PT - 1

Contact classes could work through commitment and efficacy of health managers who bear an intention to make positive change by providing them resources in the form of knowledge and/or skills; they are likely to apply these knowledge and skills in talukas where local team environment supports such change and the change agenda aligns with the local PRI and district/state expectations

37


Part 5

Mentored participants are more likely to seek opportunities to improve their local health systems to make positive change in the organisation’s performance wherever there is no hindrance (or there is an alignment) to such moves either from above or from the PRI/community structures

Elicited PT - 2

38


Part 5

Local health systems could be improved in decentralising health systems if teams have the ability to negotiate with various actors about their change proposals and if they claim decision-spaces for preparation and implementation of action plans and local decision-making at district and lower levels; if the capacity building programme could work at multiple levels to ensure better alignments between opposing elements across various actors and levels in the health system.

Elicited PT - 3

39


Part 5

40


Part 5

Macro/meso contextual factors

41

Part 5

Perceptions were aligned

42

Part 5

Receptive to technical guidance

43

Part 5

But structural problems…

44

The NRHM appointed “managers” were contractual appointeeswithin teams with very senior clinically trained doctor-managers


Part 5

“What is the use of putting my time into the PIP, if they will change it anyway at the state (level)?”

a district level health manager

“They seemed to make more noise than usual”

a senior state-level official

45

“At village level they do not really know much planning. They are actually not bothered about plans and all.”

a taluka health manager

“What do they know? After all, many of them are uneducated? What is the need for them to oversee our decisions?”


“BPMs should provide data as and when required and prepare good reports. They are too young and cannot understand the health department’s work.”


Part 5

Perceptions across the health bureaucracy

46

Narrow perceived decision-spaces

…in spite of NRHM’s on-paper decentralised planning and management since 2005

Part 5

47


Part 5

Case analysis

The hypothetical CMO frames offer a context-sensitive, theory-informed lens to analyse the intervention

– In purposively chosen talukas with and without a positive outcome (relate-able to the intervention), what were the differing contexts?

– What were the differences in the nature of commitment of the individuals in these contrasting talukas?

– ..…

48


Part 5

Explaining organisational change

• Identified case studies based on diversity of context and/or outcome after scanning context, mechanism and outcome elements

• Confronted the reformulated PT and first round of CMO-based change scenarios to these cases

49


Part 5

Case selection

• a mix of individual, organisational and contextual factors

– intervention exposure

– socio-economic development index of taluka

– mentoring interest & supervision received

– stability of team

– proxy measures of outcomes logically related to improvements in the talukas.

50


Part 5

Degree of classroom participation

(attendance and classroom activity)

(0-1.0)

Degree of mentoring received (0-1.0)

Retention of mentor inter

est by taluka High

-Moderate-Low

Organisational commitment Affective commitm

ent(AC), normative commitment (NC)

& continuance commitment (CC)

(0-5)

Self-efficacy (0-100)

Supportive degree of supervision

supervision (1-5; 1 being most

supportive and 5 being most authorit

ative)

Percentage of ever-trained

members who expressed

intention to make change

Stability of team – turnv

over (High-Moderate

-low)

Development index

Net change in percentage budge

t utilization (2008-2012)

Net change in proportion of LSCS

among total deliveries (2008-

2012)

Net change in

stillbirth rate

(2008-2012)

Gubbi 0.7 0.7 High AC 2.66

NC 2.47 CC 2.42

68 2.5 50 Mod

erate

0.95 2 1 -16

Tumkur 0.7 0.7 Mod

erate

AC 2.85

NC 2.46 CC 2.69

68 2.6 75 Low 1.21 6 1.5 -8

CN Halli 0.6 0.5 Moderate

AC 2.75 NC 2.29 CC 2.71

70 2.2 100 High 1.02 4 0.1 0

Turuvekere 0.6 0.4 Low AC 2.81 NC 2.80 CC 2.47

68 2.4 83 High 1.06 5 5.8 -4

Tiptur 0.5 0.5 Moderate

AC 2.25 NC 2.33 CC 3.17

86 2.5 75 Low 1.25 -4 12.6 -1

Koratagere 0.4 0.5 Low AC 2.87 NC 2.73 CC 3.07

71 2.3 20 Moderate

0.89 3 1.8 -3

Madhugiri 0.5 0.5 Low AC 2.50 NC 2.03 CC 2.50

83 2.4 40 High 0.82 4 1.3 -1

Pavagada 0.6 0.5 Moderate

AC 2.50 NC 2.05 CC 2.28

79 2.3 0 High 0.78 6 0 1

Kunigal 0.6 0.5 High AC 2.12 NC 2.59

CC 2.83

83 2.2 75 Moderate

0.96 2 4.9 -4

Sira 0.7 0.9 High AC 1.80 NC 2.00

CC 2.67

68 2.2 100 Moderate

0.81 6 8.3 2

51

Committed and mentored teams with low-moderate intention to make change

“In my taluka for example, I think we can make big change. It is not that everybody in my taluka want to make changes. Only one-third of them are motivated to make changes. And that is enough. I think I can make a lot of improvement by motivating these people.”

- a Gubbi taluka helath manager

“More resources mean more opportunities to make change. If they slowly give more and more power to us at taluka level, we can make many more improvements. Right now, very little is possible at taluka level. “

- another taluka health manager from Gubbi (g2)

52


Part 5

Committed health management teams could utilise new opportunities for organisational improvement presented by decentralising health systems wherever their change agenda aligns with the expectations of higher levels of the bureaucracy.

53

What PIP? What decentralisation? I sent so many requirements for staff and proposals for improvement. Only thing I got is more work, less staff and zero solutions. On one hand, I have to answer the local ZP members’ complaints and on the other hand, I have to just keep implementing plans and schemes coming from above. Nothing can be done without more staff.

- a health manager from CN Halli (cnh1)

We felt that we have to do it. So many mothers were just being referred to Tumkur. The delivery load is high and for several months, we had only one obstetrician, but somehow we managed. I know how the pressure is at the distict hospital, so having LSCS facility at Sira decreases the burden at the district hospital. It’s not easy, but somehow it is happening.

- a Sira health manager (s1)

“Nothing much can be done without giving powers at taluka level and PHCs. I cannot even appoint a Group D staff. Where is decentralisation in this?”- a PHC staff from CN Halli taluka

Tapping commitment for organisational change could be frustrating in low-resource local health systems where health managers working in poorly resourced talukas, in spite of their improved management capacities and intentions to make change, could get frustrated by the lack of facilitating action from above.

Poorly resourced teams with varying commitment levels/types & high intention for change

Discussion, relevance and lessons learned

54

Results & DiscussionResults & Discussion

Part 6

Synthesis

55


Part 6

Lessons learned – capacity building• Capacity building programmes seek to influence health

manager decisions and choices: capacity to manage alignments matter, not only determinants

• Pushing public health service organisations towards change in decentralising health organisations: need to engage with multiple levels in the bureaucracy

• Capacity building strategies need to invest more in local goal-setting and negotiation and coping skills of health managers, and not entirely focus on knowledge/skill transfer

• Capacity building programmes could seek to become the context for change through facilitating a desire for change(or harness pre-existing feeling of unhappiness with the current status) 56


Part 6

• Application of insights from organisational sciences and social sciences in health systems strengthening

• Using PT refinement and realist evaluation as an operational tool for implementation

• Need for more case diversity and further iterationscould improve the final refined theory

Lessons learned - methods

57


Part 6

Relevance• Building a human resources

management strategy for improved district health system functioning in Karnataka

• Improving the design of existing state and district level capacity building efforts

• Teaching material for teaching organisational change approaches within health servcies and for teaching theory-driven and realist approaches for evaluating healthcare interentions

58


Part 6

Thank you

59

Explaining organisational change

60

Dimensions of organisational commitment measures (AC, NC & CC – Meyer & Allen) by taluka

61
