Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Behavior Change Communication
Monitoring the process Does it work?
Webinar 4 | June 26
Assessing the situation What you need to know
Webinar 2
BCC
People eat food not nutrition Integrating BCC into nutrition programs
Webinar 1
BCC strategy and roll out The devil’s in the detail
Webinar 3
Yana Manyuk
Sight and Life Team
Expertise in social marketing & behavioral change communication 8 years experience in applying behavioral change approaches and methods to a range of projects and programs within the government, UN, and NGO organizations at HQ, regional, and field level MA from Middlesex University London in Health and Social Marketing; Distinction BA in Cultural & Media Studies - Maastricht University
Social Media
Use #SALwebinar to join the conversation!
Dr. Eva Monterrosa
Expertise in nutrition and dietetics, cultural anthropology and public health implementation science 15 years of experience in nutrition program design and assessment, formative research, social marketing, behavior change communication with academia, UN agencies, government ministries and NGOs. PhD, Nutrition, Cornell University; MS in Human Nutrition and Metabolism; BS in Nutrition and Food Science – University of Alberta
Sight and Life Team
Monitoring the Process Does it work?
Agenda
Review BCC Process Cycle Step 4, 5, 6 & 7
BCC Process Step 8: Monitoring and Evaluation
Theory of Change
BCC Objectives and Monitoring Framework
2 3 1 4
Monitoring Plan
5 6
Overview of BCC Process
Review BCC Process Cycle Step 4, 5, 6 & 7
1
3 2 4 5 6
BCC Strategy and Roll-Out
11
Checklist
Step 4. Creative Design, Step 5. Pretesting, Step 6. Training & Step 7. Implementation Completed
Strategy based on - Exchange (reducing cost & increasing short-term benefits
from perspective of the audience) - Operational mix - Content strategy
√
BCC creative brief written and shared with stakeholders √
Creative concepts & content pretested √
Training strategy is developed based on key BCC elements √
Implementation plan including costing and phasing √
BCC is a Process…
Step 1. Goals and
Objectives
Step 2. Desk
Review
Step 3. Client
Research
Step 4. Creative Design
Step 5. Pre-testing
creative
Step 6. Training
Step 7. Implement
Step 8. Monitor
and Evaluate
Webinar Objectives
Construct their own theories of change and results framework
Understand the utility of outcome monitoring for tracking results
Program managers should be able to
Learn how to set SMART objectives and what to include in a monitoring and evaluating (M&E) plan
BCC Process: Step 8 Monitoring and Evaluation
1
3 2 4 5 6
15
Continuing function that provides the management and main stakeholders with early indications of progress, or lack thereof, in the achievement of results (milestones, targets, KPIs). Monitoring is a management function – it is led and executed by the management team.
A selective exercise that attempts to systematically and objectively assess progress and achievement of an outcome (or impact). Evaluation is an evidence function – often led by an independent party not involved in the entity being evaluated.
Monitoring Evaluation
Step 8: Monitoring and Evaluation
Does one have a quality program that is functioning as intended?
a. Prerequisite for monitoring function
• Plausible program theory (tools: PIP, results chain) • Monitoring will not fix a non-existent or non-effective program • Monitoring plans are often developed shortly after the design of the
intervention is approved
b. Purpose for monitoring • Provide information on results achieved • Actionable information that yields recommendation(s) to improve
or manage better the program
c. What is monitored • Inputs, activities, outputs, and some short-term outcomes • Indicators and targets set and agreed upon by management team
d. Who is responsible for monitoring Program manager/operations team
e. How do we monitor Strategies and tools based on program needs for tracking results and ability to make changes
f. Timing • On-going indicators are reported on a monthly or quarterly basis • Comparability of indicators and data over time is critical
Step 8: Monitoring and Evaluation
Key Elements for BCC Monitoring & Evaluation
Element 1 | Theory of Change
Element 3 | Approach and Methods for the Monitoring and Evaluation Plan
Element 2 | BCC Objectives and Monitoring Framework
Step 8: Monitoring and Evaluation
Monitoring and Evaluation
Theory of Change
1
3 2 4 5 6
Theory of Change
How the intervention is expected to change behavior?
Question Tools to answer the question
What is the change process and is it plausible?
Conceptual model or behavior theory – Steps 2 & 3 Exchange – Step 4
What are the components, key functions of the intervention? Operational mix – Step 4
19
Maternal Food Choice
Physiological factors • Appetite • Pregnant or
lactating
Access to Food • Order of eating • Food sharing • Quantity &
Quality
Autonomy • Household
decision-making
• Mobility • Workload
Cognitive Factors • Knowledge • Motivations • Aspirations
Family-Related Factors • Family hierarchy
and dynamics • Household economy • Role of husband • Role of mother-in-
law
Health and Nutrition Resources Provision and uptake of services provided by frontline workers and doctors
Cultural and Economic Context (Patriarchal society, male preference, poverty)
Maternal Factors
EXCHANGE VALUE FOR HUSBAND
A small investment in food and care today = Lower risk of complication and expensive medical bills
(short-term) and a healthy baby (long-term)
EXCHANGE VALUE FOR WIFE
Small changes in food consumption today = Tangible effects on child's development
EXCHANGE VALUE FOR MOTHER-IN-LAW
Encouraging a son to be a good father and husband = Ensuring a good delivery and strong child born
What is the change process and is it plausible?
Look at exchange value
Step 8: Monitoring and Evaluation
Operational Mix Component Key Function To Whom
Promotion Media campaign Persuasion and aspiration - to motivate
Husbands, Mother-in-law, Wives
Place/Product First antenatal care receive a welcome package
Persuasion and aspiration - to motivate, to improve opportunity, improve capability
Husbands, Wives
Place/Service/Product Home visits by front line workers
Targeted education on food function Support for snacking behaviors and foods, filling treat bag - to motivate, improve capability
Wives
Support to Mother-in-law to encourage son to buy foods on grocery list - to improve opportunity
Mother-in-Law
What are the components and key functions of the intervention?
Step 8: Monitoring and Evaluation
Media Campaign
Motivations – Exchange value: 1) husbands, 2) wife, 3) MIL Increasing opportunity (access), and self efficacy (capability)
Counseling on Welcome Kit –
shopping list, SD card, letter from Chief Minister
Treat bag/box
Wife (pregnant/lactating)
Mother-in-law supports son
YOUNG HUSBAND
Champion (baby)
At first antenatal care visit
Food Function
Cards
Front Line Worker
Incr
ease
sna
ckin
g of
nut
rien
t den
se
food
s (t
wo
snac
ks fr
om a
t lea
st tw
o diffe
rent
food
s)
*Items in blue text = key interventions
Home
Theory of Change
Monitoring and Evaluation BCC Objectives and Monitoring Framework
1
3 2 4 5 6
Setting SMART Objectives
25
Our objectives were refined throughout the BCC process
Step 1. Initial Behavior Objective Women increase food intake during pregnancy
Step 3. Refined Behavior Objective Women increase food intake between meals of nutrient dense foods available in home
Step 5. Refined Behavior Objectives Women snack two times per day and choose two different snacks from seven food options
Step 8. SMART Objective At 6 months post pregnancy enrollment women are snacking once per day
SMART Objectives Specific, measurable, achievable, realistic, time-bound
BCC Objective 1 At 3 months post pregnancy enrollment, 50% husbands express intentions for buying at least 2 of the 7 foods on the food list.
BCC Objective 2 At 6 months post pregnancy enrollment, 50% of women are snacking once per day.
BCC Objective 3 At 6 months post pregnancy enrollment, 50% of women report eating at least 5 foods on the list in the previous week
BCC Objective 4 After 12 months, The Champion T.V. campaign evokes attributes (hope, pride) such as 'better future' 'better jobs' achieving more than I did’ among 75% of listeners
Increase snacking of nutrient dense foods among pregnant and lactating women (PLW)
GOAL
Step 8: Monitoring and Evaluation
Examples of SMART BCC Objectives
27
• Agree with your brain trust on the BCC objectives
• Make sure the objectives support the behavioral goals
• Objectives must focus on outcomes and not outputs
• Objectives will need indicators and these must be
included in the program monitoring plan
Tips on Setting SMART Objectives
Step 8: Monitoring and Evaluation
Inputs Activities Outputs Outcomes Impact
Results from activities – deliverables Response to stimulus
Changes observed because of set of activities. outcomes achieved during the lifecycle of the project
Benefits achieved
The action planned Stimulus
Resources available to you
Indicators that measure each domain
Monitoring Framework
Results Chain
Inputs Ac*vi*es Outputs Outcomes Impact
Impr
oved
inta
kes
amon
g PL
W
Pregnancy registration at first antenatal care visit and welcome package Intention about
snacking
Beliefs - expectancy
Fathers buy foods on food list
Welcome package counseling
Wives consume snack
Awareness for snacking
Intention to buy foods on food list
Media Campaign
Television Billboards
Emotion Media Exposure: Eligible households view campaign
Eat More, Eat Better Results Chain
Indicators that measure each domain
Self-efficacy and capacity HW PC
Media Exposure: Eligible households view campaign Parents
understand benefits of snacking
Home visits are engaging (interest)
Inputs Activities Outputs Outcomes Impact
Media Campaign
Impr
oved
inta
kes
amon
g PL
W
Front line workers visits home
Pregnancy registration at first antenatal care visit Intention about
snacking
Self esteem and engagement of HW and PC
Beliefs - expectancy
Fathers buy foods on food list
Welcome package counseling
Training of HW
Television
Encouraging/supporting Mother-in-law
Mentoring of PC
Billboards
Food & Function cards
Self efficacy – making a food choice
Emotion
Wives consume snack
Wives snack foods on food list
Treat bag (other material) is being used (interest) Control –
having a choice
Awareness for snacking
Intention to buy foods on food list
Eat More, Eat Better Results Chain
Monitoring and Evaluation Plan Approach and Methods
1
3 2 4 5 6
Results from activities – deliverables Response to stimulus
Changes observed because of a set of activities outcomes achieved during the lifecycle of the project
Benefits achieved
The action planned Stimulus
Resources available to you
Indicators that measure each domain
Step 8: Monitoring and Evaluation
Inputs Activities Outputs Outcomes Impact
MonitoringSpace Evalua*onSpace
Step 8: Monitoring and Evaluation
33
Strategy Application in BCC Main Questions Answered
Process/Implementation Monitoring
Ongoing/periodic assessment of outputs • Service delivery:
To quantify what has been done; when, where, and how it has been done
• Service utilization: Who has been reached; how the audience is reacting to the messages & activities
• Are activities being implemented according to schedule or as planned?
• What problems/bottlenecks have arisen during implementation?
• Which components of the program are or are not working?
Process Evaluation (Formative Evaluation)
Usually one-off activity, in-depth, diagnostic • Studies the domains of service utilization
and service delivery
• Completed by an independent evaluator
• How well is the program functioning?
Inputs Activities Outputs
Outcomes Impact
Step 8: Monitoring and Evaluation
34
Strategy Application in BCC Main Questions Answered
Outcome Monitoring/ Adequacy evaluation
Measures change in outcomes against BCC objectives (e.g., knowledge, self-efficacy, skills, attitudes, behaviors)
• Did the desired changes in outcomes take place?
• How much did knowledge, attitudes, and behavior change? Why?
Impact Evaluation
Benefits achieved because of the BCC intervention
Are changes in outcomes due to the BCC program?
• Did communities with the program have better results than communities without the program?
• Did people with greater exposure to the program experience better results than people with little or no exposure?
Step 8: Monitoring and Evaluation
• Quality of BCC activity implementation (process monitoring)
• The extent to which your audience practices the desired behaviors (outcome monitoring)
• Why people practice or do not practice the desired behaviors (outcome monitoring)
What to include in your BCC monitoring plan
• Throughout the project (Process Monitoring)
• At specified time points (Outcome Evaluation, based on BCC objectives)
• Who measures what, when, and where
• Budget for monitoring activities
• Quantitative Methods • (How many/What %?) • Qualitative Methods
(Why/Why not?)
1Monitoring Approach
2Timeline and Roles and Responsibilities
3Methods and Indicators
Means, Methods and Tools
1Reports & Analysis
2Participatory
3Validation
• Progress reports • Implementation plans or work
plans • Project documentation – log
frames, results chain, • Records (low technology)–
logs, inventory, patient forms, meeting minutes etc.
• Management information system (high technology environments)
• Focus groups (staff, clients) • Steering meetings • Stakeholder meetings
• Supervisory field visits • Beneficiary/client surveys
Monitoring Plan
37
Method Example Application in BCC
Inventory Tracking Tracks products/materials in stock & distributed Can help assess demand/popularity Quantity, date, and geographical locations
Event Monitoring
Uses simple questionnaire to the collect number of attendees and their feedback after an event
Bounce-back Cards Simple and short questionnaires on the back of a post-card about the intervention (e.g., How much of the leaflet did you read? Do you think it is too long?)
Audience Satisfaction Surveys
Suggestion or comment boxes, observations, qualitative interviewing/FGD’s
Social Media Monitoring Web tools that record the quantity and quality of audience interactions online provided the BCC intervention has an online component
Real-time Activity Monitoring
High quality quantitative and qualitative data collection on ongoing BCC activities through mobile technologies (e.g. receiving real time client feedback, tracking health-center performance, drop-our rates correlations)
Monitoring Methods
Monitoring and Evaluation Indicators
38
Indicator Type Application in BCC
Unaided Message Recall Measures awareness and recall ability
Understanding of Messages To what does target audience appropriately understands the promoted messages
Operationalized as clarity or confusion
Trustworthiness of BCC ideas/content
Believe in the content of provided messages or ideas.
Operationalized as belief expectancy
Intention to Adopt a Behavior Does the target audience intend to do the behavior in near future
Behavior Extent to which the target audience is practicing the behavior
Indicator Examples for Eat More, Eat Better
Engaging home visits
Awareness of snacking
Emotion
INDICATORS TARGETS MEANS FOR ERIFICATION/ WHO WILL VERIFY
I learned new information about how food affects my child from the front line worker visit today
SMS 1-yes, 2-no 75% wives in catchment households respond ‘yes’
Questions on ‘pregnant women need to eat between meals’
Exit surveys at antenatal care (ANC) Field monitors
85% of clients surveyed correctly state that women need to eat between meals
Likert scale 1 to 5: ‘I feel that my husband is more involved in my food needs’
Exit survey at ANC Field monitors
65% of women respond a 4 or more
Measuring Changes in Behavior
What can affect quality of data?
What it is: Attributing BCC outcomes to BCC intervention How to mitigate risk: Use comparison/control groups or base-line data. Triangulate data, use outcome indicators that can only be linked to your BCC intervention
What it is: Some behaviors are more difficult to adopt during certain periods of time (e.g. easting a diversified diet after harvest) How to mitigate risk: Conduct surveys at the same time of year
What it is: When people overreport positive behaviors and underreport negative behaviors How to mitigate risk: Apply good principles for effective interviewing
1Self-report Bias
2Attribution
3Seasonality
• Identify activities that are proving especially popular
• Positive feedback from audiences on specific intervention components
• New emerging national or international initiatives to capitalize on
Monitoring
41
Utilizing Monitoring results
Opportunities Threats • Intervention not
performing against expected results
• Negative media exposure • Misconception and
misinformation around the BCC intervention in community
Good management and accountability mechanisms should be in place to take action for re-prioritization of activities (opportunities) and course correction (threats) if necessary
Step 8: Monitoring and Evaluation
42
• Process monitoring should be part of your on-going management activities
• BCC SMART objectives should have clear indicators in the larger
monitoring and evaluation plan
• If you have staff monitoring, make sure they are trained to collect BCC
process monitoring. Establish an efficient system for staff to quickly report
on issues and problems with BCC activities
• Reserve a portion of overall method and evaluation budget for BCC
monitoring
• When program evaluations are conducted include BCC indicators where
appropriate
Tips on Integrating BCC Monitoring and Evaluation with Program Monitoring and Evaluation
Step 8: Monitoring and Evaluation
43
Checklist
Step 8 BCC Monitoring and Evaluation (M&E) Completed
M&E Elements discussed and approved by Brain Trust 1. Theory of Change 2. BCC Objectives 3. Monitoring Framework
√
BCC M&E Plan that includes 1. M&E Framework and Approach 2. Indicators and Methods 3. Roles and Responsibilities 4. Budget and Timelines 5. Decision and Action System
√
Overview of the BCC 8-Step Process
1
3 2 4 5 6
Overview BCC 8-Step process
Step 1. Goals and
Objectives
Step 2. Desk
Review
Step 3. Client
Research
Step 4. Creative Design
Step 5. Pre-testing
creative
Step 6. Training
Step 7. Implement
Step 8. Monitor
and Evaluate
BCC Process
46
What have we learned
1. Four Key Principles
(1) Process that culminates in a communication strategy - Systematic and strategic mix of activities and communication, NOT ONLY promotion - Only start BCC if barriers are eliminated (2) Audience focused, (3) Evidence Based - Focus on behaviors (behavior objectives) - Make no assumptions of what the audience needs and wants – ask them through research - At the heart of BCC design is the exchange. Look for insights that help you position the
desired behavior from their perspective - Turn constraints into opportunity - design around constraints - Do outcome monitoring – we are accountable
(4) Integration with program and services - Embed BCC in service delivery process – be cognizant of provider time, skills for BCC
BCC Process
47
What have we learned
3. Resource constraints Invest time and money in BCC process – not doing so is a high risk/high gamble situation. Share your experiences with us! A multi-stakeholder engagement secures buy in for the BCC process – Brain Trust Human beings are creative – unleash creative potential of your team, draw on more cost-effective resources: freelance journalist, writers, graphic designers, industrial designers
2. Iterative process Distinctions between steps – gives structure to a complex process. Look to create synergies between the steps where possible! When faced with challenges, go back to your research insights and put the audience at the center of every problem you aim to solve
Sight and Life BCC Expertise
48
Sight and Life offers expert BCC advice & services to support:
• Integrated Nutrition BCC program design, from research to implementation
• BCC Nutrition formative research
• BCC training
• Development or Co-design of Nutrition BCC related courses
• Strategic advice on demand generation within nutrition specific and sensitive
interventions
For more information contact us at: [email protected]
Join Sight and Life on the journey… for a world free of malnutrition Twitter | Facebook | LinkedIn | sightandlife.org Contact us at [email protected] Offices in Switzerland and India
Sources: IndiKit.net by People in Need. Accessed through: https://www.indikit.net/sector/1-food-security-and-nutrition Petr Schmied (2017). Behavior Change Toolkit for International Development Practitioners. Published by People in Need (PIN) Menon, S., Karl, J., & Wignaraja, K. (2009). Handbook On Planning, Monitoring And Evaluating For Development Results. UNDP Evaluation Office, New York, NY. Michie, S., Van Stralen, M. M., & West, R. (2011). The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implementation science, 6(1), 42. Rossi PH, Lipsey MW, Freeman HE. Et al. Evaluation: A systematic approach. Sage Publications: 2004. Photo credit: Michael Bloem