How to design an effective survey: The good the bad and the ugly

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Workshop

How to design an effective survey26.02.2013


How to design an effective survey

10.00-10.15 Introduction to survey design

10.15-11.15 How to design an effective surveyWriting good questions

11.15-11.30 Morning tea

11.30-12.30 Tips and tricksHow to maximise response rateGetting the most out of your data

12.30-1.00 Q&A and Close

Workshop overview



Introduction

So you want to create a survey... What is the purpose of the survey?

• Needs assessment

• Impact assessment

• Knowledge assessment

• Attitudes assessment

• Behaviour assessment

• Client satisfaction / program feedback

• Program evaluation

What survey mode will you use?• Telephone, online, paper, interview



When to use a survey

To collect information from a large number of people or records

To collect this information relatively inexpensively

To say something about the population from which the sample was drawn

When you do not need very in-depth information

When you want the data collection procedure to be as standardised as possible



Some advantages of surveys

Anonymity

Asking sensitive questions

Standardised data to monitor trends

Quick to administer

Simple data analysis (%, frequencies)

Data can be presented as graphs/charts

Multiple distribution channels (online, paper, social networking sites)



Some limitations of surveys

Limited explanations for the data

Comments open to a range of interpretations

Don’t know why questions were skipped or why survey was partially completed

No opportunity to explain questions that people don’t understand

Can’t guarantee the survey was even received or completed by the right person

Basic literacy levels needed



What makes a ‘good’ survey?

Have you ever created a survey and felt disappointed with the data you collected?

Have you ever been asked to answer a survey and felt irritated with the questions asked?

A good survey considers both your needs for data and the capabilities and experience of the respondent.

Collects complete, accurate, and reliable data.



What is a ‘bad’ survey?

Too wordy

Language is hard to understand

Bad timing - e.g. respondents tired at the end of a session

No ‘Other’ response category

Questions that seem irrelevant to the topic

Repetitive

Boring – e.g. same structure throughout, no images

Does not consider the capabilities of the respondents



Is a long survey a bad survey?


HOW TO DESIGN AN EFFECTIVE SURVEYIN 3 EASY STEPS!



Step 1.Establish your survey goals!

Ask yourself:

Why am I creating this survey?

What do I hope to learn or accomplish?

How do I plan on using the data that I collect?

How will the data influence my decisions?



Step 2. Create a ‘dummy table’

A tool you may not have used before. The dummy table helps ensure all survey objectives are addressed and all survey questions are relevant. Consider data analysis before, during and after survey development.

Areas of inquiry Questions

Demographics



A dummy table …

• Ensures you can interpret data• Ensures each question will collect the

information you need• Ensures you leave nothing of importance out• Ensures you don’t collect more information

than you need



Example: Health Care Experience of the LGBTI* Community in WA

Activity – Find questions related to:

Area of inquiry Questions

LGBTI health issues

Demographics and lifestyle

Current experience of healthcare services

How and where to reorient current health services

*LGBTI – lesbian, gay, bisexual, transgender, intersex



Answers

Area of inquiry Questions

LGBTI health issues 10,11,24

Demographics and lifestyle 1,2,3,4,5,6,7,8,9,15,16

Current experience of healthcare services

12,13,14,20,21,22,23

How and where to reorient current health services

17,18,19,25



Step 3.Write good questions

Try to use a range of question types: Open-ended questions

• To explore issues

• Open to a variety of responses (comment boxes)

Closed-ended questions• Dichotomous

• Likert scale

• Guttman scale

• Multiple choice (one response, multiple responses)

• Rank order



Example - Dichotomous



Example – Likert Scale



Example - Guttman Scale

Agreement with any one item implies agreement with the lower-order items.

E.g. are you willing to permit _______ to... - live in your country? - live in your community? - live next door to you?- marry your child?



Example – Multiple Choice (one response)



Example – Multiple Choice(multi-response)

Evaluation of Health Promotion 382



Example – Rank Order Rank the following methods of receiving a positive STI

diagnosis in order of preference from highest to lowest (1=your most preferred method; 5=your least preferred method)

• SMS______

• Phone call ______

• In person ______

• Email ______

• Letter ______



Developing a Client SatisfactionSurvey

Example: The Wheatbelt Resilience Program

Included a series of community workshops

A survey was developed to find out how useful the workshops were in terms of increasing awareness of the Wheatbelt Resilience Program; understanding of suicide and suicidal behaviours; and opportunities to help with this issue in the community



Can you identify any ways in which this feedback sheet could be improved?



The questionnaire was adapted to capture more specific information and to also gauge satisfaction with the workshop. The pre-questionnaire was on a separate document to the post-workshop evaluation.




WRITING GOOD QUESTIONS



What do you want to know?What do you need to know?

Don’t waste the opportunity to collect good quality data by asking vague questions

Be economical with the number of questions but make sure you collect all the information required



Activity: Work in small groups

Re-write these questions and include closed ended responses:

(a) “How often do you participate in physical activities on average?”

(b) “When were you last physically active?”



What to avoid

Double-barreled questions

Biased/leading questions

Questions with a double negative

Loaded questions



Example: double-barreled question

Please indicate how much you agree or disagree with each of the following statements about our service:I feel welcomed by staff and other youth at the centre

Example: leading question

Negotiating condom use with a sexual partner is difficult. Did this session help you feel more confident to negotiate condom use successfully?



Example: double negative

How do you feel about the following statement? We should not reduce investment in sexual health services.Strongly Agree AgreeDisagree Strongly Disagree

Example: Loaded questions

Which of the following services most urgently requires improvement?



What to avoid

Questions which could create a response seti.e. the tendency for a respondent to answer later questions the same way they answered earlier questions

Technical language/jargonUse simple, common words

Emotive languageDo you approve of the government’s oppressive immigration policy?



What to avoid

Using “and”, “or”, “but” in the same question

Long, complex, multi-part questions

Questions which contain a negative and a positive

If you were in a foreign country and did not speak the language but knew someone who did would you…

a. Let the person that spoke the language speak for you

b. Try to learn the language on your own

c. Not try to learn the language or let the person that you knew that spoke the language speak for you.

Huh?!



What to avoid

Response categories that are not exhaustiveWhat is your current age? 10 to 20 21 to 30 31 to 40

Response categories that are not mutually exclusiveWhat is your current age?

10 or less 10 to 20 20 to 30 30 or greater

I’m 41!

I’m 30!



Avoid fuzzy words in the question

Those words which are open to different interpretation by the respondents.

• Have you ever had any major injuries during the past year?

Avoid words like: important, major, better, regular, often, occasional (subjective interpretation).

Best to give specific time & details



Irrelevant questions

Questions which are irrelevant to the survey objectives

• E.g. do you really need to know someone’s income?

Questions which are irrelevant to the respondent• e.g. Don’t ask nurses to compare the effectiveness of different

patient management techniques if they have no experience of the alternatives.



Presuming knowledge

Don’t assume people either know or have appropriate experience to answer the question.

• Have you been bullied this year?

• How many standard drinks have you drunk in the last week?

• How many cigarettes have you smoked in the last week?

Use filter questions or provide more information to avoid these problems

What is bullying?

I don’t smoke!

What is a standard

drink?



Presuming knowledge

Hypothetical or predictive questions• Would you like to try marijuana?

• Avoid - many people might like to try anything once.

Generally avoid:• Would you like to...

• What would you do if..?



Personal questions

Be sensitive!

May need to preface personal questions to give respondent ‘permission’ to respond truthfully

Consider the structure of response choices• people don’t like to consider themselves at extreme ends of a

scale



Activity: Design a brief survey to evaluate this workshop

Areas of inquiry Questions

Demographics

Workshop objectives

• Increase participants’ knowledge and skills in effective survey design• Assess participants’ satisfaction with the SiREN training workshop


TIPS AND TRICKS



Critically assess your surveyquestions Is the respondent’s understanding of response

categories likely to be influenced by more than words?

The vaguer the question and answer categories, the greater potential for error / poor data

Is the survey information being collected in more than one mode? E.g. phone, interview, email



Critically assess your surveyquestions

Is changing a question acceptable to the survey sponsor?

Is the question necessary?

Will it improve the validity of your study if you do ask more than one question?



Critically assess your surveyquestions

How easy is it to get an accurate answer? Can people accurately recall and report past

behaviours? Will the respondent be willing to reveal the

requested information? Will the respondent feel threatened or defensive

by the question? Will the respondent feel motivated to answer each

question?



Tips and tricks

Insert dummy questions• These may be irrelevant to the research to avoid respondents

guessing desirable answers

Reverse the wording of some survey items• This technique can avoid creating response sets

I always enjoy going to work

I dread going to work



Tips and tricks

The order of questions may influence responses• Put interesting questions first• Sensitive questions towards end• Demographic information – location of these questions depends

on importance• Similar themes together

Put instructions for questions before the question not at the beginning of the survey



Tips and tricks

Be specificExample: How can we improve our service? This is too vague and may elicit single responses about many different aspects of the service without much guidance to help you decide which ones are important.Instead, consider asking respondents to rank from 1-5 the quality of different aspects of the service

What claims will you use the data to support?Be honest and consider if this is realistic or credible. This technique helps you to clarify the wording of questions


HOW TO MAXIMISE RESPONSE RATE



Aim to get a 65% response rate

That is: 65% of your sample complete and return a questionnaire

Remember non-respondents may bias your results

Choose a sample size that takes into account response rates

Sample size for pre- and post- surveys should be large enough in case of attrition

If possible, have participants complete survey at the time of distribution



How to maximise response rate

Involve your target group in design if possible Pre-test!

Ask questions that people can and will answer

Uplift, motivate and encourage the respondents to give information

Avoid bad survey questions

Allow sufficient time for responding to the survey and for participants to complete the survey



How to increase responses

Plan multiple contacts• Pre-notification (tell people you are planning to survey)

• Reminders for non-responders

Consider incentives• Individual incentives more effective than prize draws e.g. incentive

given for getting others to complete the survey

• Dependent on resources

Obtain ethics approval for your survey



A note about survey ethics…

Considerations may include:

Information and consent forms

Confidentiality and anonymity

Samples which include minors

How to ask questions about potentially illegal activities

Length of survey / burden on respondents

Format of survey - e.g. acceptability, culturally appropriate

Please contact [email protected] for more advice about survey ethics

mailto:[email protected]


GETTING THE MOST OUT OF YOUR SURVEY DATA





Planning and evaluation cycle

Needs assessment

Program planning

Implementation

Evaluation

Re-planning Use different types of survey data as part of a continuous program planning and evaluation cycle

Survey data

Survey data

Survey data Survey

data



How will you use the data?

Use survey data to inform the program planning cycle

Adopt a mindset of continuous program improvement

After implementing the survey, critically evaluate the quality and completeness of the data collected

• Were any questions ambiguous?

• Were all the questions necessary?

• Did any questions appear to cause confusion?

• Did you get some unexpected responses?

• Were the data useful?

• What decisions could the data inform?

• Are there still gaps in the data?

Ask yourself what will you change?



Close the feedback loop

Finally, it is important to ‘close the loop’ when collecting data by following up respondents to thank them for their input and to share the results of the survey

If you intend to survey the same group several times, this feedback loop is particularly important and may influence the response rate



Getting the most out of your data

Don’t just file the survey data away until the next activity report for your funding provider is due!



Health promotion short course – example survey data

3

10

3

Pre course

Not

c

onfid

ent

Confid

ent

Very

conf

ident

0

79

Post courseHow confident do you feel to contribute to the design and implementation of an evaluation of a health promotion initiative?

How confident do you feel to undertake a needs assessment to plan a health promotion program?

Not

c

onfid

ent

Confid

ent

Very

conf

ident

2

11

3

Pre course

Not

c

onfid

ent

Confid

ent

Very

Confid

ent

0

5

11Post course



What could you conclude from these data?

What decisions about the course would you make as a result of these data?

Are these decisions valid? What other data might you want to collect?

Health promotion short course – example survey data

How confident do you feel to work collaboratively with your community and with other organisations?

Not

c

onfid

ent

Confid

ent

Very

Confid

ent

0

13

3

Pre course

Not

conf

ident

Confid

ent

Very

conf

ident

0

4

12

Post course



Any questions?


FOLLOW UP SESSION



Follow up session

In two weeks’ time!

Aim to put what you have learned into practice

Either:1. Critically review a survey you have used previously OR

2. Design a new survey using what you have learned.

Bring your surveys and any questions to the survey clinic

SURVEY CLINICTuesday 12th March 2013, 10am

– 12 noonCurtin University (Room TBA)



Workshop evaluation

Please draw around your hand (or both hands) and provide some feedback on today’s workshop.



THANKS FOR COMING!

Please don’t forget to complete the workshop evaluation! Thank you

Education

How to design an effective survey: The good the bad and the ugly