Embed Size (px)
Citation preview
Program Evaluation : Quantitative Vs. Qualitative
Approaches
IndiaCLEN Program Evaluation Network
Narendra K. AroraDepartment of Pediatrics
All India Institute of Medical SciencesNew Delhi
Program Evaluation- Chandigarh 29th February 2004
Research is not being translated into policy
“ The dominant finding of our review is that there are large gaps between the care people should receive and the care they do receive. This is true for all types of care - preventive, acute, and chronic [...] for different types of health insurance […] for all age groups, from children to the elderly […] ”
Schuster et al, Milbank Quarterly, 1998
Evidence
Research vs. Policy & Practice
Research FOR policyPriority setting for health researchProgram designProgram monitoring and evaluationPractice guidelines; Rational use of drugsNeeds assessment: introduction of new technology & programs
Research ON policy Privatization of health servicesClinical Audit
Research and Policy & Practice
Policy for ResearchPromotion of Research
Practice (based) on ResearchEvidence based practice
Research and Policy & Practice
Natural Passive Diffusion of Information
Research
Publications
Changes in Policy & Practice
Information alone is insufficient for changes in policy and practiceResearchers are not systematically involved in implementation of their own findingsLack of policy relevant evidence
Scientific evidence vs. need of policy makersGuidelines
Not commensurate with resultsNot understandable by Policy makers
Research and Policy & Practice
Policy Driven Research
Research
Priority Issues
Policy & Practice
Implement
Research
Implement
Policy & Practice
Conventional Research
Components of a Program
Policy: Needs & RationalePolicy: Aims & GoalProgram: StrategyPlanning: Implementation
Program Evaluation
• A systematic effort to describe the status of a program
• Extent to which program objectives achieved
GovernmentProgram Managers Policy Makers
Funding AgenciesHealth Care Research InstitutionsCliniciansCommunity
Identifying the stakeholders
Consumers of Evaluation Research
Use of Program Evaluation Data
Policy Makers / Program managers– Needs Assessment– Redefining aims / objectives – Modifying or fine tuning strategies (process)– Sustainability (including fatigue factor)– Judge the worth (impact)– Expense / cost– Interaction with other activities / health systems
Needs of Policy Makers
Academia
- Wider application of program strategies / Cross country / culture
– Unique features (success/failure)– Determinants of provider and client behavior
Use of Program Evaluation Data
Types of Evaluation
Needs Assessment
To identify
• Goals
• Products
• Problems
• Conditions
Types of Evaluation Contd…
Formative (Process) Evaluation
To improve developing or ongoing program
Role as helper/advisor/planner
Progress in achievements
Potential problems/needs for improvements in
• Program Management
• Inter-sectoral coordination
• Social mobilization
Implementation
Outcomes
Types of Evaluation Contd…
Summative (Coverage) Evaluation
(To help decide ultimate fate)
Summary statement about
Program’s achievements
Unanticipated Out comes
Comparison with other programs
Qualitative Vs Quantitative Research - the dichotomy
Qualitative Quantitative
Social theory Action Structure
Methods Observation, Interview
Experiment, Survey
Question What is X ?
(classification)
How many Xs?
(enumeration)
Reasoning Inductive Deductive
Sampling method Theoretical Statistical
Strength Validity Reliability
Qualitative Research: Key Strengths
•Qualitative methods aim to make sense of, or interpret, phenomena in terms of the meanings people bring to them
•Qualitative research may define preliminary questions which can then be addressed in quantitative studies
•A good qualitative study will address a clinical problem through a clearly formulated question & using more than one research method (triangulation)
•Analysis of qualitative data can & should be done using explicit, systematic, & reproducible methods
Potential limitation of Qualitative Methods
• Accuracy of the information• representative nature• Cultural inappropriateness• Subjectivity of the Investigator
Qualitative Research Methods
Findings
- Presented alone / in combination with quantitative data
- Validity and reliability depends on methodological skills, sensitivity, integrity of the researchers
- Skillful interviewing - more than just asking questions
- Content analysis - more than just reading to see what’s there
- Generate useful and credible findings through observation, interviewing and content analysis
How?
- Discipline, knowledge, training, practice, creativity, hard work
Vit-A/IFA Program Evaluation, 2001-02:An IndiaCLEN Study
Quality Assurance Measures* Common understanding of
the objectives and data collection methods
* Group consensus in finalizing the interview schedules
Level 1National Protocol Finalization Workshop
(CCT+SI)
* Supervise Focus Group Discussion & its transcription and translation
Level 3 CCT Member Field visits (FGDs)
* Supervision in 30% interviews by SI* Schedules countersigned by SI* Tape recording of interviews* Cross checking of transcripts and translations * Quality Assurance RA
Level 4 Data Collection
* Common understanding of the objectives and data collection methods
* Hands on experience in the fieldLevel 2 Regional Workshops
(CCT+RC+PMC/NGOs)
* A few (CCT Members) are involved who are Trained and Experienced
* Group consensus
Level 6Data Analysis & Report Writing
(CCO)
* Random check of 10% taperecorded interviews
Level 5 Data Management, CCO
Multidisciplinary Central Coordinating Team
Program Evaluation
Program Evaluation Expert
Epidemiologist
Health Social Scientist
Biostatistician Anthropologist
Economist
Clinicians
Health Program Evaluation - Quantitative Research Methods
Approach
- Measures the reaction of a great many people to a limited set of questions
- Comparison and statistical aggregation of the data
- Broad, generalizable set of findings presented succinctly and
parsimoniously.
Multi-Centric Evaluation Studies
• Cohesive network of partners• Common understanding of the program• Common understanding of aims & objectives of evaluation exercise• Standardization of research instruments• Standardization of protocol
implementation at various sites• Regional variation in program
implementation• Multiple layers of quality assurance
measures• Generalizable and hence more confidence
Challenges in Delivery of ‘Routine Public Health Programs’ in India:
Vitamin-A and Iron folic acid Supplementation Program(s) - a case study
Supported by
Ministry of Health & Family Welfare Government of India,
The Micronutrient Initiative, Canada,
INCLEN, USAID & AIIMS
Characteristics of A Routine Program?
“We are not bothered about those we could not reach because they are not deficient in Vitamin-A. Otherwise they would have come with night blindness in OPD.”
District level provider (8001): Kolkata
“Those who come to us we give them. Clients not coming must be healthy.”
Government health worker (1006): Ballabgarh
“We are already over burdened. What can we do there?”
Government health worker (11013): Nagpur
Passiveness in Implementation
Resistance to Polio Drops among Clients
Rumours• Rationale and safety of repeated dosing not clear• Rumors regarding vaccine safety (sterility/HIV-AIDS)• AFP cases occurring in children who had received
OPV drops in the past
Who are Resistant/Reluctant Clients
"We heard that the only purpose (of giving these drops to our children) is that they cannot produce children in future."
Client (Non-utilizer Men)
"Cold (OPV) drops will make them (children) cold (no sexual arousal). People believe in rumors that children will become impotent."
Client (Non-utilizer Women)
Social Mobilization Strategy• Use of force – counter productive/‘fatal’
“Police came and they forcibly gave us the drops.”
Client (Non-utilizer Men)
Resistant/Reluctant Clients
“I have seen it myself that force was not used on Hindus but when they (health workers) approached Muslim houses, they used force to give drops to their children.”
Client (Non-utilizer Men)
“There is no need to use force (for good things like polio drops). If you try to explain by compassion, people will understand (accept it). Suppose you give me something to eat and I refuse to oblige, if you pick up a stick and force (threaten) me to do it, it invites suspicion that something is fishy about it and that is why I am being forced to eat. If you offer with love, I will take it. This problem exists everywhere.”
Client (Male Non-utilizer)
"If a Mohammedan doctor or a priest (Maulvi) comes and explains, people will understand and give the drops. Otherwise people will think that government is asking us to drink poison (drink drink....... )."
Client (Non-utilizer Men)
“This is an unprecedented event where all people irrespective of caste, creed and religion take part in PPI program on the same day (NID) throughout the country”
•Health worker (150): Burdwan
“He (my husband) told me that everybody is going for polio drops. Then why should we be left out ? After all, everybody is not a fool”
•Utilizer (1422): Delhi
Quotable Quotes
• Give a vivid, meaningful flavor which is far more convincing than pages of summarized numbers
- These should not be distracters
- Should not take the reader away from the real issues in hand
Reasons for Non-utilization of FHAC Program Services: Process Evaluation
Reasons
Provider perspective
NGO/ leader
Perspective
Client perspective
Lack of awareness about FHAC 1+ 1+ 4+
Low motivation 2+ 2+ 1+
Shyness / Embarrassment 2+ to 3+ 2+ 1+
Adversities (facilitatory factors) 1+ <1+ <1+
Client Conveniences
* Season / month
* Distance
* Timing of camp
<1+
1+
2+
1+
1+
2+ to 3+
1+ to 2+
1+
3+
Camp facilities 1+ 1+ 1+
Low credibility 1+ <1+ <1+
Semi Quantitative Qualifiers: 1+ Some/a few, 2+ About half, 3+ A majority, 4+ Most, 5+ Almost all
IndiaCLEN FHAC Program,2000 : Process Evaluation
Reasons for Non-participation
Men
% (95% CI)
Women
% (95% CI)
Total
Not aware about FHAC program
82.3 (79.9-84.8) 79.7 (76.9-82.5) 81.0 (78.7-83.4)
Indifferent 2.4 (1.6-3.4) 2.6 (1.9-3.4) 2.5 (1.9-3.1)
Embarrassed 0.3 (0.0-0.6) 1.3 (0.8-1.8) 0.8 (0.5-1.2)
Not aware about camp
0.8 (0.4-1.3) 0.7 (0.3-1.0) 0.7 (0.4-1.0)
Need not felt 2.3 (1.6-3.0) 5.6 (4.4-6.8) 3.9 (3.2-4.7)
Inconvenient timings 2.0 (1.1-3.0) 2.4 (1.2-3.5) 2.2 (0.5-2.9)
Loss of wages (affordability)
8.0 (6.5-9.4) 5.8 (4.5-7.3) 7.0 (5.8-8.1)
Not in village (availability)
1.7 (1.0-2.3) 1.6 (1.0-2.1) 1.6 (1.2-2.1)
Reasons for Non-participation in FHAC Camps by the Clients(RURAL)
IndiaCLEN FHAC Program,2000 : Coverage Evaluation
What is a Safe Injection?• All injections are safe
• Right drug/ good medicine/ expiry of drug has been checked/ right dose/specific drugs are safe/skin sensitivity test should be done before givinginjection/ drug dissolve in muscles/drug should be tested/injection should be given with glucose/ single dose vial to be used
• Use new plastic syringes/ syringe & needle from a sealed pack / expensive syringe/needle/boiled syringe and needle/sterilized glass syringes & needle/one needle and syringe for one patient/ use of disposable syringes / needles / ISI mark/avoid reuse chemical stylizations of syringes of syringes & needle/ new needle from the sealed pack
• Needle and syringe – clean / proper/ flushing of syringes and needle using spirit / water/steel needles.
• Technique of giving injection: gives slowly/properly/painless injection/proper site of giving injection/giving intramuscularly is safe
What is a Safe Injection?• Before injection ask patient to roll up shirt/ clean the injection site with spirit swab• Injection giver wash hands/ use gloves/observe sterile precaution• Injection taken from recognized hospital • Injection given when necessary/ rational use• Disease should get completely cured/ maximum curative effect• Prevention of disease by immunization• Injection that prevent blood borne diseases• Does not cause reaction• Needle and syringe flushed & cleaned before disposal/ syringe thrown away• Doctors know about safe injection• Safe to reuse syringes/ needles to the same person• No injection is safe• Clients should be made aware about the safe injection
Conti ….
When do you (Prescriber) give an injection?
• Initial stages of diseases, for quick relief• Patient not improving with oral drug, For serious diseases• Side effects of oral drugs• Doctor decides, Prescriber prefer injection • Doctor wants to earn money• Type of disease• In case of emergency• Better compliance to injection• To restore energy• Patient demand Injection• Certain categories of patients (poor, uneducated, rural patient)• Those who can afford injection• Vaccines and immunization• No preference- both oral & injections are equally effective• Supply is more/ date of expiry is near
73%
2%7% 11% 7%
Not aware about FHAC
Aware but not attended camp
Contacted at home alone
Attended camp only
Contacted at home & attended camp82%
1%3% 9% 5%
Reach of Family Health Awareness Campaign Program
Rural Areas
Urban Slums
IndiaCLEN FHAC Program,2000 : Coverage Evaluation
Results
Qualitative Vs. Quantitative
• It is not qualitative vs. quantitative but qualitative and quantitative
• Mechanism and dynamics of events captured by qualitative methods
• Range of possible answers expanded through qualitative methods and then relative significance determined by quantitative methods
• Combination of methods make the results as well as recommendations more meaningful and operational
Concluding Remarks
Evaluators express their opinions explicitly
- Based on evidence gathered
- Consistent quality assurance measures
- Limitations of methods accepted up front
- Politics of evaluation
- Remains a scientific endeavor
- Efforts to overcome barriers
(keep in mind the perspectives & perceptions of the stakeholders)
•
•
•
•
•••
Srinagar
Rohtak
AgraLucknow Dibrugarh (2)
Berhampur
Bhopal
Bilaspur
ChennaiVellore
Nagpur
Mumbai (2)
Gulbarga
Bangalore
Calicut Coimbatore
Thiruvananthapuram
•
•Bijapur
Burdwan•Patna•
Jaipur •
Kangra•
Chittorgarh• Gwalior• • Darbhanga
Kolkata (4)•
••Guwahati
Kohima
ImphalAizwal
SambhalpurCuttack
• •
Vijayawada
Hyderabad (2)•
Kannur••
Jamnagar
Jodhpur•
Dehradun••Chandigarh
Udaipur•Rajkot
•Surat
Pune
•Manipal
Belgaum
Gangtok
Ranchi
Bhubaneshwar
Varanasi
Jabalpur•
•
••
Visakhapatnam
•
•
Amritsar•
Aligarh•Ghaziabad•
•
Muzaffarpur
Gaya
••
Shillong• Agartala•
•
Ajmer•Barmer•
Bhuj•
Bhavanagar•
Baroda• Raipur•
•
Panaji ••
ThirunelveliMadurai•Thrissur•Kottayam•
• Kakinada
•Kurnool
Bhagalpur
Jamshedpur••
Network Partners(n==84)
An IndiaCLEN Program Evaluation Network 2003-04
Medical Colleges 69
NGOs 9
Pub Health Inst 6
IndiaCLEN Program Evaluation Network
[IPEN]
VISION
Facilitate development
and implementation of
people friendly, effective
Public Health Programs
that are in harmony
with their
socio-cultural beliefs,
thus creating a milieu
where service providers
get motivated and
communities consider
Public Health Services
beneficial and
willingly participate
