Analyzing your Survey Data: The Impact of the Campaign

KAP Data analysis Part 2: BR, BC, TR and CR

By the end of this lesson you will be able to:

• Analyze your data according to your analytical plan (BR, BC, TR, CR)

• Analyze your data based on your reporting needs

• Complete Sections 3.4 B – D of your Campaign Learning Report that identify the impact of your campaign along the Theory of Change

Structure for the Day

• Check in 9:30 – 10:00 AM

• Individual Work time 10:00 – 12:00• Check in 1:30 – 2:00 PM• Individual Work time 2:00 – 3:00 • Check in/Assignment 3:00 – 3:30

4

Assessing Impact

• Provides an overview of the results of the campaign along the:

–BR SMART Objectives

–BC SMART Objectives and

–TR SMART Objectives

–CR SMART Objectives

Flow/Guidelines

1. Review of your KAP Monitoring Plan (BR, BC, TR, CR) and Hypothesis

2. Complete the tables in your section 3.4 per Target Audience

3. Write a short narrative after each table describing how well your campaign achieved its SMART Objectives

4. Assess the impact of your campaign along TOC

Analyze Data: Statistics clarify truth

Show critical data in body of report (SMART Objectives)– Tables– Charts

Show all your data in tables in Appendix:– Honest thing to do– May show you where your campaign failed so you can fix it in

future!

Analyze Data: Comparability of Surveys• Best:

– All Chi-Square tests < 95%• OK:

– A few Chi-Square tests ≥ 95%– Differences in frequencies small (5 to 10 percentage

points)• So-so:

– Several Chi-Square tests ≥ 95%– Differences in frequencies large (10 to 20 percentage

points)• Unacceptable:

– Many Chi-Square tests ≥ 95%– Differences in frequencies large (> 20 percentage

points)

Worse Case Scenario• Assume you have large differences in gender:

– Baseline = 35% male– Post-campaign = 56% male– Difference of 21 percentage points

• Problem: If a dependent variable increases by 15 pp, it could be due to 2 things:– Pride campaign impact– More men in 2nd sample

• Solution is to “control” for gender using filters:– Filter for men, run analysis– Filter for women, run analysis

Reminder # 1

• For BR, BC, TR and CR, KAP SMART objectives are based on self-reported data.

• Triangulate results with non-KAP metrics

Analyzing BRExample: BR KAP Question(36) I am going to read you a number of statements about the management of the local no-take area. For each statement, I would like you to tell me if you strongly agree, agree, disagree, or strongly disagree with it.

(E) There is enough money and other resources to fully manage and enforce the rules of the no-take area

[ ] SA[ ] A[ ] D[ ] SD[ ] NS/DK(G) The rules of the no-take area are unclear and local fishers don't understand them[ ] SA[ ] A[ ] D[ ] SD[ ] NS/DK

BR: Non-KAP Measure?

Example: BC Question

(43)During the past 6 months, would you say that you have been regularly involved, occasionally involved, or not involved with the creation and/or the management of a no-take fishing area in your local area

(A) [ ] Regularly involved [ ] Occasionally involved [ ] Never involved [ ] Don't know / not applicable

Analyzing BC

BC: Non-KAP Measure?

Example: TR Question

(41) I am going to read you a list of different types of fishers, and for each one, I would like you to tell me whether you remember seeing someone like that fishing in this area in the past 6 months (show the NTZ on a map of the area but don't mention whether it is NTZ or not)(A) Subsistence fishers from your village

[ ] Seen[ ] Not seen[ ] Not sure / Don't remember (B) Subsistence fishers from nearby villages[ ] Seen[ ] Not seen[ ] Not sure / Don't remember

Analyzing TR

TR: Non-KAP Measure?

Example: CR Question

Has your catch increased, decreased or stayed the same as a result of the Lola Marine Sanctuary? (If the person does not fish or glean mark as NA)

[ ] Decreased[ ] Increased[ ] Stayed the Same[ ] N/A

Analyzing CR

CR: Non-KAP Measure?

Structured Time:KAP monitoring Plan data survey template• Proceed to section 3.4A in your

campaign learning report

SMART Objectiv

e

ToC Category

  

Metric

(KAP or

non-KAP)

Pre-campa

ign result

Pre-campai

gn frequency error(if KAP)

Target

Post-campa

ign result

Post-campai

gn freque

ncy error

(if KAP)

Chi-square

d significance (if

KAP)

Change(in pp

if applica

ble)

   Knowledge 

             

   Attitude 

             

 

 Interpersonal

Communication

 

             

Reminder # 2

ToC SMART Objectives Metric Method

Baseline (Pre-campaign)

Result (Post-campaign)

Change (in percentage points)

Chi-Square (X2) Significance

SMART Objective Attainment

Interpersonal communication GENERAL COMMUNITY

By September 2010, 25 % of Southern Residents who do not hunt will have spoken with someone about wildland fires in the past 6 months (a 10 percentage point increase from 15%; N=237).

Q28: In the past 6 months, have you talked to anyone about wildland fires?

KAP Survey Analysis

15 34 19 >95significant 190%

1.GENERAL COMMUNITY (NON-HUNTERS)

Interpersonal CommunicationOur goal for interpersonal communication was simple. We wanted people to talk to each other about wildland fires, about the causes, the effects, the billboards, etc. We achieved 190% of the objective for the general community, though the data for the hunters was not statistically significant and cannot be assessed. This may be attributed to a very small sample size of this audience in the pre campaign survey.

Respondents were also asked if they had heard that there were wildland fires in Guam’s watersheds. While cannot be linked directly to interpersonal communication, it is an interesting question to look at to see if there is an increase in people hearing about fires since the pre campaign survey. The general results of all of the respondents showed an increase from 22% to 45% of people who had heard about wildland fires, indicating that there is indeed more information about fires being communicated.

While many of the campaign images focused on the impacts of wildland fires, such as the campaign poster and billboards, they did not directly state that fires were caused by people and that Guam had a very low occurrence of natural fires. The campaign display game addressed this as did the community and school presentations, but it may have been advantageous to include this messaging into more campaign materials. It may also be a good idea to do some supplemental surveys with more direct questions about these important knowledge concepts to get a more accurate representation of what people know.

Reminder # 2

• Use the exact SMART objectives languageEx. Strongly agree is not the same with agree

SMART Objective

TA1 - Fishers

The number of Burgos/Uba local fishers who says strongly agree that the Burgos Marine Sanctuary regulations need to be followed will increase from 86.9% measured in February, 2011 to 89.9% in July, 2012 ( an increase of 3pp, Q30F in the KAP survey)

SurveyPro result

Scenario

• There is an increase in reported BC but the SMART Objective target are not met.

WHY?

Could it be how you set your SMART target?

• Historic data• Baseline - Diffusion of innovation• Type of audience

Þ Increase/DecreaseÞ Maintain

The potential for change for different types of objectives across the Theory of Change

22

Criteria K A IC BC

Average Percent of Target Audience Changed

22pp 13pp 28pp 14pp

Sample Size (n of campaigns) 213 139 42 45

Historic Data

Pride campaign average results summary till 2010

Baseline and diffusion of innovation

According to Diffusions of Innovation the Rate of Change Depends on the Starting Point

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30

Time (Relative)

% A

dopt

ion

of In

nova

tionn

Innovators – 2.5%

Early Adopters – 13.5%

The Late Majority – 34%

Laggards – 16%

The Early Majority – 34%

Source: Everett Rogers, graph from Wikipedia.org

The potential of change for different types of audiences

24

Baseline\Target Audience (data based on median of knowledge)

General Public

Influencer Resource User

<20% 20pp 24pp 1.7pp20% to 40% 23pp 37pp 25pp40% to 60% 31pp 33pp 21pp>60% 17pp 8pp 16pp

• Selective perception(Hassinger) – people who don’t “want” to know don’t seem to learn.

• The critical mass phenomenon / social norms

3. Using right words for accuracy

The SMART objective should use the same words as the question

& as the answer option used

Reminder # 3

• Check if you use the right filter. The latest appears in the dropdown menu. Take note of the code of the filters you use.

Stages of behavior question

Can it be used to identify the current stage ofbehavior?

Reminder # 5

• Try other figures beside using tables

• When you are done with your analysis, do a publish report

Reminder # 6

• When you are done with your analysis, do a publish report