Designing Your Survey. By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey. Differentiate between a biased and

Designing Your Survey



By the end of this lesson

you will be able to:

• Define the 6 goals of the

questionnaire survey.

• Differentiate between a

biased and an unbiased

survey question.

• Develop survey questions

from preliminary

objectives.


• Design a survey that:

— has appropriate sample

sizes and strategy

— measures the impact of

your Pride campaign on

SMART objectives for each

target audience

— measures exposure to

your Pride campaign

activities


Learning to Detect Bias and Poorly written

questions Reading Activity Writing the best possible survey questions

SESSION TOPICS AND TIMING

Creating Questions from Preliminary Objectives Drafting Your Stages of Behavior Questions Determining Sample Sizes and Calculating

Your Sites Sample Size

Assignment, Rubrics and Research Plan

Review

Thurs

Friday


Activity 1: Learning to Detect Bias and Poorly Written Questions (Basics)

PURPOSE: To practice recognizing “bias” on questions and identify what

makes a question biased

45 minutes


Revisit the Theory of Change


INSTRUCTIONS:1. Read the Designing Your Survey Lesson Content (pp 321-336) – 20

minutes

2. Proceed to your workshop group (3-4 persons per groups) and answer

the guiding questions in pages 295-298 – 10 minutes

3. Record your groups responses and report your results (5

minutes)

ACTIVITY 2: READING


Why Conduct an Interview Survey

• Surveys are research tools that provide

quantitative information about people in your

target area.

• The systematic sampling will make your

respondents “representative” of the population

of interest.

• Surveys are ideal tools to measure change in

your target audiences because you can measure

the “baseline” and post-campaign levels.

— The difference between baseline and post-

campaign survey results provides an estimate

of the impact of your campaign on that

variable.


Questionnaires and your Pride Campaign

• You will implement 2

surveys during your

campaign:

— During the planning

phase (baseline)

— At the conclusion of

your camp


Goals for your survey

1. Describe your Target

Audience(s)

“ 90 % of all fishers

surveyed are male and 60

% of those are over the

age of 40”


Goals for your survey– pre campaign

2. Assign Respondents to

their Appropriate Stage-of-

Behavior-Change

“Among fishermen surveyed, 5

percent are currently setting

rattraps on their boats, 5

percent are currently thinking

about setting rattraps...”



3. Establish a Baseline for your SMART

Objectives

“30% of Fishers know that rats get to Serena on

fishing boats” can be the basis for setting the SMART

Objectives:

“By 2013, the percentage of Fishers who know that

rats travel to Serena on their boats will increase from

30% to 50%


Goals for your survey– post campaign

4. Measure the impact of your PRIDE campaign

in achieving your SMART Objectives

ICInterpersonal communications

AAttitude

KKnowledge

Control site


5. Measure

the Audience

Exposure to

your Pride

Campaign

and

Attribute

Effects to

your Pride

Campaign

CARE FOR THE LASTING PRIDE

A campaign to protect Tagbilaran City’s underwater kingdom




6. Ground Truth – test

your assumptions about

community, strategy,

and ToC

"99% of community members

know about the importance of

keeping rats off of the island”…

what does this tell you

about your ToC?


DESIGNING YOUR QUESTIONNAIRE


A Note of Caution…

Developing and implementing

quantitative surveys is a lot of

work • bias questions

• ask questions of the wrong people, or at least not enough

of the “right” ones, of your critical target audience(s)

• ask too many questions

• make mistakes in choosing your samples - your baseline

and post-campaign samples are not comparable

• make mistakes during data entry you enter the wrong

data into the wrong variable and end up with meaningless

results


Biased Questions

The most important challenge of

survey questions is that they can be

BIASED:

Biased questions mean: they make respondents

tend to pick one response over another because of

the way the question is worded or the way the

response options are offered.


Biased Questions

Biased questions:

• Try to educate

• Assume what they ask (are leading)

• Are double-barreled, asking 2 questions at

once

• Are ambiguous or confusing


Key Elements of Good Survey Questions?

Unbiased – not leading or

pushing respondents towards a

specific answer Tied to Objectives Time bound Clear Not Judgemental


Writing good survey questions

Other important criteria for good

survey questions:

• Include a relevant time frame

• Use filter questions as needed

• Don’t overlap (e.g. age ranges)

• Use appropriate language for the respondents


Activity 3: Writing the Best possible Survey Questions

Purpose:

To Practice writing “good” survey questions and appropriate response

choices

Instructions: (Turn to pages 299-310)

1. Work in pairs

2. Read through each of the survey questions assigned to you and

the corresponding responses found

3. For each of the question, do the following:

a. Identify what makes this as poorly written questions


Survey Questionnaire #1

Original Survey Question/Poorly Written Question

Do you know what the Ministry’s law on legal size of fish that can be killed is about?

YES NO Not Sure

Rewritten Survey Question

Does the Ministry have a law about the legal size of fish catch? If yes, please tell me what you think the legal size of fish catch is?

YES NO Not Sure




Do you agree that people should no longer slaughter elephants just for their tusks?

YES NO Not Sure


I am going to read you a statement and I would like for you to decide if you strongly agree, agree, disagree, strongly disagree or if you have no opinion: “Should people be allowed to kill elephants for their tusks”?

Strongly Agree

Agree No Opinion Disagree Strongly Disagree


Survey Questionnaire #3Original Survey Question/Poorly Written Question

For what reasons have you gone into the Bungo Protected Area? Tell me all that Apply

Hunting NTFP Farming Other


I am going to read you a list of possible reasons that people sometimes go into Bungo Protected area. I would like you to tell me for each reason whether you have gone into the Bungo PA for that reason during the past 6 months.Reasons Yes No Not Sure

Hunting

Gathering plants for medicines or food

Farming

Any other reason, please specify




Have you purchased bushmeat?YES NO Not

Sure


Have you purchased or bartered for bushmeat, or meat from the wild animals, at any time in the past 6 months?

YES NO Not Sure




How many times would you say that you have gone into the Jamu Protected Area?

Never

1 to 4 times

4 to 10 times

10 t0 20 times

Rewritten Question

In the last six months, how many times would you say that you have been in this area here (show them the map of the protected area)

Never

1 to 4 times

5 to 10 times

More than 10 times




Have you talked to anyone about Jamu?

YES NO Not Sure


In the last 6 months, have you spoken to anyone in your community about the importance of protected Jamu for the health of the people who live here?

YES NO Not Sure




What are the threats to the Jamu Ecosystem?

Fire

Drought

Poaching Develop-ment

Rewritten Question

What types of activities and conditions threaten the health and sustainability of important wildlife in Jamu?

Fire

Drought

Poaching

Develop-ment

Other




In the past 6 months, have you ever fished illegally?

YES NO Not Sure


In the last 6 months, have you ever used dynamite fishing?

YES NO Not Sure




Do you think that rats can spread the disease typhoid fever to humans and cause damage to nets?

YES NO Not Sure


I am going to read you to you a series of statements about rats. Please tell me if you think they area true, false or you are unsure:

Rat spread typhoid fever to humans.Rats cause damage to fishing nets

True False Not Sure


Survey Questionnaire #11Original Survey Question/Poorly Written Question

Do you agree that overfishing is a problem and that something needs to be done about it?

YES NO Not Sure

Rewritten Survey QuestionI am going to read you to you a series of statements and I would like you to tell me whether or not you strongly agree, agree, disagree, strongly disagree or have no opinion:1. Overfishing is a problem2. The government should create

laws to prevent overfishing3. The Banga community should

support a ban on overfishing

StronglyAgree

SAAgree

A

Disagree

DStrongly Disagree

SD

No Opinion

NO


YOUR QUESTIONS?


1. Read and review the lesson content

especially on Biased and Unbiased

questions (pages 326-336)

2. Read Lesson Content pages 337-348.

Write in a piece of paper 2-3 questions (per

participant) that you would want to be

clarified or know more.

READING ASSIGNMENTS FOR TOMORROW


REQUIREMENTS:

• Identified Preliminary Target

Audience

• Preliminary SMART Objectives

DESIGNING YOUR QUESTIONNAIRE


How Many QUESTIONNAIRES?

ONLY ONE QUESTIONNAIRE (Pre and

Post KAP Survey) for all target

audiences Some questions though may not be

applicable to some audience Create a system where some

questions are answered by certain

target audience while others are not.


How Many QUESTIONS should I ask?

Factors that can affect your survey

are: the number of respondents length of questions who will administer Data entry Data analysis and writing Cost

Include questions that will answer

only the six goals of survey


Where do the survey questions come from?

KEY CHARACTERISTICS1. Date

2. Enumerator Name

3. Form #

4. Geographic Area/ Sampling location

5. Respondent Information: Age, gender, education level, income

sources, religion?

6. Knowledge Question

7. Attitude Question

8. Interpersonal Communication Question

9. Behavior Change Question

10.Stages of Behavior Change

11.Barrier Removal (As Appropriate)

12.Exposure to Materials


SMART Examples


Behavior Change and Questionnaire SurveysStatement from stage of behavior

change question in surveyStage of

behavior change it should

correspond to

I have never considered using rat traps in my fishing boat.

Validation stage

I am considering using rat traps in my fishing boat but haven’t decided yet.

Pre-contemplation stage

I intend to use rat traps in my fishing boat in the near future, but I haven’t talked to anyone about it yet.

Maintenance stage

I have talked to someone about my intention to use rat traps in my fishing boat.

Action Stage

I used rat traps in my fishing boat at least once. Contemplation stage

I always use rat traps in my fishing boat. Preparation stage


Sample Behavior Change Questions

Clearly define the “desired behavior

change” of your campaign Make one question each along the Stages

of Behavior (Behavior Change continuum) Example:..\..\EXAMPLE behavior

change questions.docx


Exposure Questions

Make at least 8-10 Examples of potential

materials that you will be producing in

your campaign and ask the participants if

they have see See example

..\..\EXAMPLE media exposure questions.docx


TOP TIPS

1. Complete each interview in about 30 minutes or less

2. The categories should also be “exhaustive;” that is,

there is no other possible response that a respondent

wants to give but cannot.

3. Be sure that your questions directly relate to what it

is you are studying, especially for those questions

intended to measure SMART objectives.


Top Tips…..

4. Be very aware of the fact that people in your

target communities may not have the same

vocabulary as you.

5. If you want to force people to choose from a

short list of possible responses, use the closed-

ended format.

6. If there are some questions that you feel might

be sensitive you should put these at the end of the

survey so that respondents don’t get upset at the

beginning.


Top Tips

7. Have one person do the first translation from

the language the survey was drafted in into the

second language. Then, have a different person

translate the survey back into the original language

(back translation).

8. “Administer the survey to a small number of

respondents (five is usually enough) similar to your

target population to make sure that categories are

clear and complete”.


Key Elements of Good Survey Questions?

Unbiased – not leading or

pushing respondents towards a

specific answer Tied to Objectives Time bound Clear Not Judgemental


Writing good survey questions

Other important criteria for good

survey questions:

• Include a relevant time frame

• Use filter questions as needed

• Don’t overlap (e.g. age ranges)

• Use appropriate language for the respondents


Writing good survey questionsFILTER QUESTIONS

In the past 6 months, have you talked to anyone about benefits to

the community of a well enforced sanctuary? YES NO

If you have, please tell me all of the people with whom you have

talked to about this. [ ] Have not talked to anyone

[ ] Talked to spouse/partner

[ ] Talked to parents, or in-laws

[ ] Talked to your children aged 16 or older

[ ] Talked to your children aged 15 or younger

[ ] Talked to friend or neighbour

[ ] Talked to village elder or local authority

[ ] Talked to National government environmental officials

[ ] Talked to Non-government environment officials

[ ] Talked to Local government environmental officials

[ ] Talked to a Local XXXX Fisher or Gleaner

[ ] Other ________________


Writing good survey questions RESPONSE QUESTIONS TO

CATEGORIES SHOULD NOT OVERLAP

PROVIDE ACCURATE TRANSLATION

CLOSE VERSUS OPEN-ENDED

QUESTIONS


ORDER OF QUESTIONS

Use transitions between modules –

use to switch from one topic to

another Provide Accurate translation Review and Pretest your

Questionnaire


Criteria for a Good Survey Questionnaire

• Uses the Rare template as a starting point, and asks the

socioeconomic and demographic questions exactly as are

in template.

• There is a clear link to the campaign’s objectives so that

the data can help in campaign design, test assumptions

and be used to measuring its impact across all stages of

the Theory of Change.

• Questions cover most important factors such as:

— Socioeconomic and Demographic profiles

— Stages of behavior change question

— SMART objectives measurement

— Exposure to Campaign Activities & Messages


Criteria ….

• Questions are clear, unbiased and appropriate level of

language and vocabulary for the target audiences.

• Interpersonal communication, behavior change, and

exposure to Pride activity questions are time-bound, if

appropriate, so you can measure change, normally this

means they state a time of less than 1 year.

• Questionnaire did not touch upon socially sensitive issues

which introduces bias.

• Questionnaire takes not more than 30 minutes to

administer to a respondent.

• Questionnaire was pretested.


INDIVIDUAL/ GROUP WORKTIMEDrafting your Survey Questions

Your Draft Questions must follow the

template provided to you earlier Add:

Behavior Change Questions Media Exposure Questions


Conducting the Survey


Guide to Individual/Group Work

(60 MINUTES) – 10:00-11:00

1. Review the Serena Island Research Plan (tables 5&6) and pages

312-313 in your packet

2. Using your preliminary Results Chains and preliminary SMART

Objectives begin to develop your site's survey questions. Take the

time to develop survey questions

3. Incorporate the newly developed survey questions into the

Quantitative Data Collection section of your site's Research

Plan. (Section 5.4 & 5.5)

(30 MINUTES) 11:00-11:30

4. Share with another CF within your group

5. Consult/ Share with your PPM on your Survey Questions


Other Considerations

1. Filter Questions

2. Response Categories for Questions Should

not Overlap

3. Closed versus Open-ended Questions

4. Order of questions

5. Provide accurate translation

6. Review and pretest your questionnaire


Selecting Your Sample

1. Representative sample

2. Sample size

3. Identify your random sampling

procedure


ASSIGNMENT

ASSIGNMENT #7, Draft Quantitative

Survey (ComX 5337) Points: 30 points DATE ASSIGNED: 21 September 2012 DATE DUE: 24 September, Monday,

9:00PM What to Submit:

o Draft Survey Questions for K, A, IC,

BC, Stage of Behavior Questions

and Media Exposure


Grading CriteriaPoint

sTied to SMART objectivesAlthough there are a lot of questions that you might be interested in asking your respondents, the goal of your questionnaire survey is to understand your impact and exposure in relation to your objectives. Ensuring that the survey question is clearly tied to your objectives will make understanding your impact transparent.

10

UnbiasedThe questions are not leading or pushing the respondent towards a specific answer.

5

Follows logical orderThe survey should be organized so that simple and non-sensitive questions come first with any sensitive questions coming last. Also, survey should test knowledge of topics before describing what those topics are. Questions should flow in a logical order.

5

ClearIf the questions are complex or unclear, your responses will be as well! Simple questions that are not overly complicated by jargon will make responses to your survey precise and accurate.

5

Not judgmental or emotionalOverly emotional survey questions will make respondents feel uncomfortable and may answer in a way that doesn’t accurately reflect how they feel.

5

Total 30Penalty for late submission (-1.5)


Revisiting Your Research Plan Template

SECTION 5.0 QUANTITATIVE DATA

COLLECTION

5.1 KAP Survey

5.2 Sampling Design

5.3 Survey Planning Table

5.4 Draft Survey Questions

5.5 Draft Stage-of-Behavior

Questions


By the end of this lesson

you will be able to:

• Define the 6 goals of the

questionnaire survey.

• Differentiate between a

biased and an unbiased

survey question.

• Develop survey questions

from preliminary

objectives.


• Design a survey that:

— has appropriate sample

sizes and strategy

— measures the impact of

your Pride campaign on

SMART objectives for each

target audience

— measures exposure to

your Pride campaign

activities

Documents

Designing Your Survey. By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey. Differentiate between a biased and