Upload
jehill3
View
1.791
Download
0
Embed Size (px)
DESCRIPTION
Preliminary results from a survey on the use of metrics and evaluation strategies among mHealth projectsPatricia Mechael, Nadi KaongaCenter for Global Health and Economic Development at the Earth Institute, Columbia UniversityCORE Group Spring Meeting, April 30, 2010
Citation preview
Preliminary results from a survey on Preliminary results from a survey on the use of metrics and evaluation the use of metrics and evaluation
strategies among mHealth projectsstrategies among mHealth projects
Garrett Mehl, Franz Allmayer, Heli Bathija Department of Reproductive Health and Research, WHO, Switzerland
Patricia Mechael, Nadi KaongaCenter for Global Health and Economic Development at the Earth Institute, Columbia University
Survey Instrument• 50 multiple choice and open ended questions
• ~ 15 minutes for each project
• Survey topics
• Health focus areas, strategic approaches
• Specific objectives, types of monitoring, approach to evaluation
• Project phase, duration, level of support
• Self-identified areas of need
Survey Methodology
• Jointly developed with Columbia U.
• The survey is still being carried out.
• Available as a web-survey or soft-copy for printing
• Survey was announced to various electronic forums
• Additionally, individuals were sent targeted emails.
Survey ReachSurvey Reach
USA:5
Peru: 7
Haiti:2Mexico: 3
India:6
Pakistan:4Kenya:11Uganda:10South Africa:4Tanzania:5Ghana:3Nigeria:5Malawi: 2
Philipines:4
All countries highlighted contain an mHealth project covered by the survey. Countries with multiple projects are specified.
Jordan
Project duration in sample
Projects by health focus
Current Phase of projects
Current Phase
Needs assessment 8.8%
Usability testing 8.8%
Pilot not for scaling 7.4%
Pilot for scaling 57.4%
Large scale implementation
17.6%
Frequency of project objectives
• Client information
• Health provider information
• Increasing access to services
• Improve service quality and/or safety
• Increase client service demand
• Increase time savings
• Improve intra-provider communication
• Increase provider skills
• Reduce unneeded referrals
• Increase patient treatment compliance
• Reduce health service costs
• Improve ability to respond to crises
• Reduce stock-outs
Client focusProvider focusEfficiencies
mHealth Strategies
Clustering of mHealth objectives
StockCrisis
response
Time savingsReduce referral
Communication
Provider Skills
Lower costsQuality/
SafetyCompliance
Service DemandService access
Client info
Provider info
Domains of Measurement
Measurement domains of focus
Level of monitoring
Focus on Monitoring and Evaluation
Focus of Evaluation Assessment
Did the intervention result in improvements in:
Number of evaluation questions tracked
CostsSustainabilityBehavior changesHealth outcomesKnowledge, attitudesPerformanceQuality of CareService Utilization
Type of evaluation approach
Descriptive 29.8%
Cross-sectional 44.7%
Longitudinal 40%
Case-control 12.8%
Wedge 8.5%
Rigor of Evaluation Design
Drivers of Monitoring
Level of Monitoring
Significance
Performance Accounting
correlation coefficient
= .392(spearman’s
rho)
sig. (2-tailed)
.002
Number of funding sources
correlation coefficient
= .456(spearman’s
rho)
sig. (2-tailed)
.000
Project phase and monitoring level
Evaluation rigor by project start date
Evaluation rigor by project phase
Drivers of evaluation rigorRigor of
evaluationSignificance
Performance Accounting
correlation coefficient
= .501(spearman’s
rho)
sig. (2-tailed)
.000
Number of funding sources
correlation coefficient
= .392(spearman’s
rho)
sig. (2-tailed)
.001
Assistance requested
Assistance need by project phase
“We need a systematic approach to analyzing the
data we have collected over the past 3 years. “
Assistance Requested
“We also need to learn what is the norm for "success" in this field and
how we stack up to normal interventions vs. other mHealth
projects working on [similar] technology.”
“We need guidance on evaluation methods for
mHealth”
“We are interested in collaborative approaches
and standard indicators that will be measured across the
different mHealth programs.”
“How to assess the impact of [our] mHealth tool.”
Thank you.
For more information, or to submit your mHealth project to the survey, please send an
email to:
Dr. Garrett [email protected]