1 1 Designing Surveys for Research and Evaluation © Fraser Health Authority, 2007 The Fraser Health Authority (“FH”) authorizes the use, reproduction and/or

11

Designing Surveys for Research and Designing Surveys for Research and EvaluationEvaluation

© Fraser Health Authority, 2007

The Fraser Health Authority (“FH”) authorizes the use, reproduction and/or modification of this publication for purposes other than commercial redistribution. In consideration for this authorization, the user agrees that any unmodified reproduction of this publication shall retain all copyright and proprietary notices. If the user modifies the content of this publication, all FH copyright notices shall be removed, however FH shall be acknowledged as the author of the source publication.

Reproduction or storage of this publication in any form by any means for the purpose of commercial redistribution is strictly prohibited.

This publication is intended to provide general information only, and should not be relied on as providing specific healthcare, legal or other professional advice. The Fraser Health Authority, and every person involved in the creation of this publication, disclaims any warranty, express or implied, as to its accuracy, completeness or currency, and disclaims all liability in respect of any actions, including the results of any actions, taken or not taken in reliance on the information contained herein.

Your fortune…Your fortune…

Workshop OutlineWorkshop Outline

Identifying Information Needs

Question Design

Data Coding and Analysis

Survey Sampling and Sample Size

Data Collection Methods

What is a survey?What is a survey? SurveySurvey

A method or process of gathering information from a sample of A method or process of gathering information from a sample of individuals.individuals.

The physical format or tool employed to gather information: The physical format or tool employed to gather information: Opinion poll, questionnaire, customer satisfaction, voting, Opinion poll, questionnaire, customer satisfaction, voting, census etc.census etc.

Why are surveys used?Why are surveys used? 1. To answer a question.1. To answer a question.

• Program Evaluation- Did we deliver the services that we planned to deliver?Program Evaluation- Did we deliver the services that we planned to deliver?

2. To gain information on a population or topic.2. To gain information on a population or topic.• Patient Satisfaction.Patient Satisfaction.

3. To gather objective data to make informed decisions.3. To gather objective data to make informed decisions.• Priority setting.Priority setting.

4. To provide a benchmark.4. To provide a benchmark.• "snapshot" in time, establish a baseline to measure change."snapshot" in time, establish a baseline to measure change.

When to use a Survey?When to use a Survey?

When questions and information needs When questions and information needs are best answered by the people are best answered by the people themselves.themselves.

When information is not available from When information is not available from existing sources.existing sources.

So You Would Like to Design a So You Would Like to Design a Survey…..Survey…..

Will need to complete Will need to complete some basic survey some basic survey planning steps.planning steps.

Identifying Information NeedsIdentifying Information Needs

1) State the problem.1) State the problem. 2) Identify parameters related to problem.2) Identify parameters related to problem.

Who, where, when, what?Who, where, when, what? 3) Identify concepts to be measured.3) Identify concepts to be measured. 4) How much detail for each item.4) How much detail for each item. 5) How will the results be presented.5) How will the results be presented.

Statistics Canada (2005)Statistics Canada (2005)

Survey Sampling and Sample Survey Sampling and Sample SizeSize

FrameFrame• The physical means that you will use to access the The physical means that you will use to access the

information for the population.information for the population.

Phone book, map, telephone numbers, address, Phone book, map, telephone numbers, address, department, hospital ward, employee listing etc.department, hospital ward, employee listing etc.

Example: Program registration forms.Example: Program registration forms.

Survey PopulationSurvey Population The survey population is the population that is The survey population is the population that is

covered by the frame. covered by the frame.

Example, all 10,000 educational program Example, all 10,000 educational program recipients.recipients.

Now that you have determined who will Now that you have determined who will comprise your survey population, it is time to comprise your survey population, it is time to think about sampling.think about sampling.

Probability Sampling MethodsProbability Sampling Methods

A A probability samplingprobability sampling method is any method method is any method of sampling that utilizes some form of of sampling that utilizes some form of random random selectionselection. .

Must set up some process or procedure that Must set up some process or procedure that assures that the different units in the population assures that the different units in the population have have equal probabilitiesequal probabilities of being chosen. of being chosen.

Sampling: Things to Keep in MindSampling: Things to Keep in Mind

Probabilistic or random sampling methods Probabilistic or random sampling methods generally preferred over non-probabilistic ones, generally preferred over non-probabilistic ones, because they are more accurate and rigorous. because they are more accurate and rigorous.

If you want to make accurate generalizations to If you want to make accurate generalizations to the population regarding your sample, you will the population regarding your sample, you will need to use need to use probability sampling methods.probability sampling methods.

If you select your sample by convenience, the If you select your sample by convenience, the conclusions drawn from the survey results apply conclusions drawn from the survey results apply only to the only to the SURVEY SAMPLE. SURVEY SAMPLE.

E.g. Non-probability sampling methods.E.g. Non-probability sampling methods.

Non-probability methods are acceptable Non-probability methods are acceptable providing the method reflects the purpose of providing the method reflects the purpose of why why you are surveyingyou are surveying, and will give you the , and will give you the information you are looking for.information you are looking for.

Sampling: Things to Keep in Sampling: Things to Keep in MindMind

Probability Sampling Methods: Probability Sampling Methods: RandomRandom

There are several methods to choose from:There are several methods to choose from: Simple random Simple random

sampling.sampling.

Probability Sampling Methods: Probability Sampling Methods: StratifiedStratified

Stratified samplingStratified sampling (divide the (divide the

population into non-overlapping population into non-overlapping

strata and sample from within strata and sample from within each stratum independently).each stratum independently).

Guarantees representation of all Guarantees representation of all important groups.important groups.

Selection of the sample using an interval “k” so Selection of the sample using an interval “k” so that every “k” unit in the frame is selected, is that every “k” unit in the frame is selected, is called called systematic systematic

random samplingrandom sampling..

Probability Sampling Methods: Probability Sampling Methods: SystematicSystematic

Steps to achieve a Steps to achieve a systematic random samplesystematic random sample: :

1. Number the units in the population from 1 to N.1. Number the units in the population from 1 to N.2. Decide on the n (sample size) that you want or need. 2. Decide on the n (sample size) that you want or need.

k = N/n = the interval size. k = N/n = the interval size. 3. Randomly select an integer between 1 and k. 3. Randomly select an integer between 1 and k. 4. Then take every kth unit. 4. Then take every kth unit.

Example: Example: 1.1. N=200N=2002.2. n=40, take N/n, 200/40=5 (interval size).n=40, take N/n, 200/40=5 (interval size).3.3. Randomly select a number between 1 and 5 (let’s pick 4).Randomly select a number between 1 and 5 (let’s pick 4).4.4. Begin with 4, and take every 5Begin with 4, and take every 5thth unit. unit.

Probability Sampling Methods: Probability Sampling Methods: SystematicSystematic

Probability Sampling Methods: Probability Sampling Methods: ClusterCluster

Cluster sampling.Cluster sampling. Divide population into clusters and Divide population into clusters and

randomly sample clusters. randomly sample clusters. Measure Measure allall units within sampled clusters. units within sampled clusters. Example: See blue areas on map. Example: See blue areas on map. Not just geographic areas, could Not just geographic areas, could select hospitals, schools select hospitals, schools etc. etc.

In social research, most people use a In social research, most people use a combination of these sampling methods.combination of these sampling methods.

Called Called multi-stage sampling.multi-stage sampling. Tailor your sampling method to the Tailor your sampling method to the

purpose of your survey.purpose of your survey.

Probability SamplingProbability Sampling

Non-Probability Sampling MethodsNon-Probability Sampling Methods There are different types of non-probability There are different types of non-probability

sampling methods as well:sampling methods as well: ConvenienceConvenience (not representative of population). (not representative of population). PurposivePurposive (certain group in mind). (certain group in mind). Expert samplingExpert sampling (seek out specific expertise). (seek out specific expertise). Snowball samplingSnowball sampling (ask people to participate, they (ask people to participate, they

ask more people).ask more people).

If you select non-probability sampling If you select non-probability sampling methods, the conclusions drawn from the methods, the conclusions drawn from the survey results apply only to the survey results apply only to the SURVEY SURVEY population.population.

Sample Size: How many?Sample Size: How many? Considerations:Considerations:

Level of precision desired (when generalizing to population, the Level of precision desired (when generalizing to population, the larger the sample, the greater the precision).larger the sample, the greater the precision).

Size of population.Size of population. Budget.Budget. Time.Time. Sampling method.Sampling method. Response rate.Response rate. Type of survey.Type of survey.

Good general guideline: Sample size table.Good general guideline: Sample size table.

Remember…Remember…

If you sample the entire population, you If you sample the entire population, you do not need to worry about statistical do not need to worry about statistical tests, or generalizing your findings.tests, or generalizing your findings.

Example: CensusExample: Census All cases are represented, can speak All cases are represented, can speak

with 100% confidence about entire with 100% confidence about entire group.group.

Data Collection MethodsData Collection Methods Now that we have Now that we have

decided on sampling decided on sampling methods, we need to methods, we need to think about the way in think about the way in which we collect the which we collect the information.information.

What type of data What type of data collection method will collection method will you choose?you choose?

Methodological Considerations for Methodological Considerations for SurveysSurveys

It is possible to incorporate experimental and It is possible to incorporate experimental and quasi-experimental designs in survey research.quasi-experimental designs in survey research. Link back to purpose of survey.Link back to purpose of survey.

Do you need to have a control group?Do you need to have a control group? Do you need to use probability sampling?Do you need to use probability sampling? Will you have multiple data collections?Will you have multiple data collections?

Will the same individuals participate at both times? Will the same individuals participate at both times? How will you identify these individuals?How will you identify these individuals?

Data Collection MethodsData Collection Methods

Face to face interview. Face to face interview.

Exit survey (ex. Leaving mall).Exit survey (ex. Leaving mall).

Telephone interview. Telephone interview.

Self-completed questionnaire.Self-completed questionnaire.

Web-based survey.Web-based survey.


Data collection methods.Data collection methods. What type of method will you select?What type of method will you select?

Considerations.Considerations. Cost.Cost. Time.Time. Accessibility and type of population (are mail Accessibility and type of population (are mail

or telephone an option?).or telephone an option?). Amount, type, and quality of data required.Amount, type, and quality of data required.

Method of Method of data data collectioncollection

CostCost TimeTime Response Response raterate

LengthLength ComplexComplex

FACE TO FACE TO FACEFACE

HIGHHIGH MEDIUMMEDIUM HIGHHIGH LONGLONG HIGHHIGH

EXITEXIT MEDIUMMEDIUM FAST-FAST-SLOWSLOW

MEDIUMMEDIUM SHORTSHORT LOWLOW

TELEPHONETELEPHONE MEDIUMMEDIUM FASTFAST MEDIUM-MEDIUM-HIGHHIGH

MEDIUMMEDIUM MEDIUMMEDIUM

MAILMAIL LOWLOW SLOWSLOW VERY VERY LOWLOW

MEDIUMMEDIUM LOWLOW

WEB-BASEDWEB-BASED LOWLOW FASTFAST LOW-LOW-MEDIUMMEDIUM

SHORT-SHORT-MEDIUMMEDIUM

MEDIUMMEDIUM


Once you have decided Once you have decided on the data collection on the data collection method, it is time to method, it is time to think about the content think about the content of your survey.of your survey.

What are the What are the questions questions that will comprise your that will comprise your survey?survey?

Question Development: TypesQuestion Development: Types

Open-ended questions:Open-ended questions: Respondents are free to Respondents are free to express answers in own words.express answers in own words.

Closed-ended questionsClosed-ended questions: Respondents must : Respondents must choose responses from a list.choose responses from a list.

Partially-open questionsPartially-open questions: Respondents can : Respondents can choose from categories provided, or compose choose from categories provided, or compose their own answers.their own answers.

Open-ended QuestionsOpen-ended Questions

Used often in qualitative research (i.e. Used often in qualitative research (i.e. focus groups, case studies).focus groups, case studies).

Advantages:Advantages: Provide opportunity for self-expression.Provide opportunity for self-expression. Can obtain exact numerical data if requiredCan obtain exact numerical data if required

• Example: How many times have you been to the Example: How many times have you been to the hospital in the last year?hospital in the last year? 1616

Can obtain natural wording.Can obtain natural wording.

Open-ended Question ExampleOpen-ended Question Example

Hello Sir, can you tell us about the service Hello Sir, can you tell us about the service you received at the grocery store today?you received at the grocery store today?

““My experience at your My experience at your store was horrible! The store was horrible! The employees were rude, employees were rude, you did not have what I you did not have what I wanted, and I got a wanted, and I got a ticket because my ticket because my parking meter ran out of parking meter ran out of time!”time!”

Open-ended QuestionsOpen-ended Questions Limitations for Limitations for

respondentrespondent::

• QuestionsQuestions are are demanding.demanding.

• Time consuming.Time consuming.

Limitations for Limitations for researcherresearcher::

• Can yield irrelevant Can yield irrelevant answers.answers.• Difficult/time Difficult/time consuming to code and consuming to code and analyze.analyze.• Expensive.Expensive.

Closed-ended QuestionsClosed-ended Questions

Very common in surveys.Very common in surveys. Advantages: Advantages:

Easy to answer.Easy to answer. Fast to answer.Fast to answer. Easy to code.Easy to code. Easier and faster to analyze.Easier and faster to analyze. Less expensive (survey processing and Less expensive (survey processing and

analysis).analysis). Consistent response categories.Consistent response categories.

Closed-ended Question Closed-ended Question ExampleExample

BinaryBinary:: Were you satisfied with the service you Were you satisfied with the service you

received?received? YesYes NoNo

Multiple ChoiceMultiple Choice:: How long have you worked for the clinic?How long have you worked for the clinic?

Less than 1 year.Less than 1 year. 1 to 5 years.1 to 5 years. 6-10 years.6-10 years. More than 10 years.More than 10 years.

Limitations:Limitations:

More effort in design stage.More effort in design stage. May elicit answer where no option/knowledge May elicit answer where no option/knowledge

exists.exists. Response categories must be exhaustive and Response categories must be exhaustive and

non-overlapping.non-overlapping.

Closed-ended QuestionsClosed-ended Questions

Allows respondent to choose an option, BUT also gives Allows respondent to choose an option, BUT also gives them the opportunity to choose “OTHER, PLEASE them the opportunity to choose “OTHER, PLEASE SPECIFY”.SPECIFY”.

Example:Example: Were you happy with the service you received at the Were you happy with the service you received at the

diabetes clinic?diabetes clinic?

YesYesNoNoNot SureNot Sure

If you selected No, why was this the case? If you selected No, why was this the case? __________________________________________________________________________________________________________

Partially-Open QuestionsPartially-Open Questions

Rating ScalesRating Scales

Using the scale below, please rate our staff on Using the scale below, please rate our staff on the following items:the following items:

1=Poor1=Poor2=Fair2=Fair3=Good3=Good4=Excellent4=Excellent

Knowledge 1Knowledge 1 2 2 3 3 44Friendliness 1Friendliness 1 2 2 3 3 44Promptness 1Promptness 1 2 2 3 3 44

Commonly Used Rating ScalesCommonly Used Rating Scales

Satisfaction scalesSatisfaction scales (very dissatisfied to very satisfied).(very dissatisfied to very satisfied).

Agree/Disagree scalesAgree/Disagree scales (strongly disagree to strongly agree).(strongly disagree to strongly agree). Universal scale: Can be applied to all ratings.Universal scale: Can be applied to all ratings.

Performance scalesPerformance scales (poor to excellent).(poor to excellent). Can be used widely, very good for overall ratings.Can be used widely, very good for overall ratings.

Frequency scalesFrequency scales (never to very often).(never to very often). Not very common, but can sometimes be used.Not very common, but can sometimes be used.

Common Satisfaction ScalesCommon Satisfaction Scales 2 point satisfaction2 point satisfaction

1-satisfied1-satisfied 2-unsatisfied2-unsatisfied

4 point satisfaction4 point satisfaction 1-very satisfied 1-very satisfied 2-satisfied 2-satisfied 3-dissatisfied3-dissatisfied 4-very dissatisfied4-very dissatisfied

5 point satisfaction5 point satisfaction 1-very satisfied1-very satisfied 2-satisfied2-satisfied 3-neither dissatisfied or satisfied (neutral) 3-neither dissatisfied or satisfied (neutral) 4-dissatisfied4-dissatisfied 5-very dissatisfied5-very dissatisfied

Common Satisfaction ScalesCommon Satisfaction Scales

Also 7 point scales (have a Also 7 point scales (have a somewhat satisfiedsomewhat satisfied and and somewhat dissatisfiedsomewhat dissatisfied categories)…use if categories)…use if applicable for your particular purpose.applicable for your particular purpose.

1-very satisfied1-very satisfied 2-satisfied2-satisfied 3-3-somewhatsomewhat satisfied satisfied 4-neither dissatisfied or satisfied (neutral)4-neither dissatisfied or satisfied (neutral) 5-5-somewhatsomewhat dissatisfied dissatisfied 6-dissatisfied6-dissatisfied 7-very dissatisfied7-very dissatisfied

Can Can mean mean

similar similar things...things...

Common Performance ScalesCommon Performance Scales

4 point excellence 4 point excellence scale.scale.

1-excellent1-excellent 2-good2-good 3-fair3-fair 4- poor4- poor

5 point excellence 5 point excellence scale.scale.

1-excellent1-excellent 2-good2-good 3-fair3-fair 4-poor4-poor 5-very poor5-very poor

Can also have expectation scales:Can also have expectation scales:

4 point expectation.4 point expectation. 1-exceeded1-exceeded 2-met2-met 3-nearly met3-nearly met 4-missed4-missed

Can also have importance scales (very to not Can also have importance scales (very to not very important).very important).

Common Performance ScalesCommon Performance Scales

Frequency ScalesFrequency Scales

Use when you have questions related to how Use when you have questions related to how often.often. NeverNever RarelyRarely SometimesSometimes OftenOften AlwaysAlways

E.g. How often are you kept waiting at your doctor’s E.g. How often are you kept waiting at your doctor’s office?office?

Factors to consider:Factors to consider:

1) Select the 1) Select the appropriate scaleappropriate scale!! Match scale to question. Find the most natural scale Match scale to question. Find the most natural scale

through informal pre-testing.through informal pre-testing.

E.g. I can easily get the information I need to do my E.g. I can easily get the information I need to do my job well.job well.

Would you use a frequency scale (how often), or an Would you use a frequency scale (how often), or an agreement scale (strength of agreement)?agreement scale (strength of agreement)?

Rating Scales: Factors to ConsiderRating Scales: Factors to Consider

2) 2) DirectionDirection Doesn’t matter as long as it’s clear to Doesn’t matter as long as it’s clear to

respondent.respondent. Do not change scale direction, keep one Do not change scale direction, keep one

direction for all questions.direction for all questions.


3) 3) Number of choicesNumber of choices: there is no specific : there is no specific number, pick the one that works best for you.number, pick the one that works best for you.

• Large scales (10 pt +), harder to answer and label.Large scales (10 pt +), harder to answer and label.

• Shorter scales not as sensitive.Shorter scales not as sensitive.

4) 4) Label CategoriesLabel Categories instead of using only numbers instead of using only numbers Will make distinction between categories clear.Will make distinction between categories clear.

5) 5) MidpointMidpoint Will yield more information than forcing a Will yield more information than forcing a

pro/con response. pro/con response.

E.g. Neither dissatisfied or satisfied (neutral).E.g. Neither dissatisfied or satisfied (neutral).


6) Don’t know, not applicable, and neutral are 6) Don’t know, not applicable, and neutral are differentdifferent..

Important to consider these options when Important to consider these options when appropriate.appropriate.

Don’t KnowDon’t Know: respondent lacks knowledge to : respondent lacks knowledge to make judgment.make judgment.

Not ApplicableNot Applicable: respondent cannot relate to : respondent cannot relate to statement.statement.

NeutralNeutral: respondent has come to middle of two : respondent has come to middle of two extremes.extremes.


7) Response set.7) Response set. Respondents tend to repeat previous answers in Respondents tend to repeat previous answers in

rating questions.rating questions. Long series of rating questions should be broken Long series of rating questions should be broken

up.up. Insert other question types between rating Insert other question types between rating

scales.scales.


Visual Analogue ScalesVisual Analogue Scales

Visual Analogue Scale (VAS) is a measurement Visual Analogue Scale (VAS) is a measurement instrument that tries to measure a characteristic instrument that tries to measure a characteristic or attitude that is believed to range across a or attitude that is believed to range across a continuum of values and cannot easily be continuum of values and cannot easily be directly measured (Gould, 2001).directly measured (Gould, 2001).

Example:Example:

Considerations for WordingConsiderations for Wording

• Keep your audience in mind.Keep your audience in mind.• Are the questions easy to understand?Are the questions easy to understand?• Do the questions have the same meaning to Do the questions have the same meaning to

all?all?• Define important terms.Define important terms.• Limit biasLimit bias

Bias – Question Design Wording

Ambiguous/inappropriate question• Were you happy with the service you received at the diabetes clinic?

Complex question• Are you in favour, or not in favour of a law that would not allow store

to be open on Sundays nor stat holidays?” Double-barrelled question

• If you watch TV regularly, what kind of shows do you watch? Technical jargon

• Do you agree that IH should have access to SPSS or SAS to support quantitative survey analysis?”

Uncommon word• Assist, reside, sufficient, deleterious etc.

Vague word• Regularly, generally, (can also apply to emotional or value based

words – e.g. feel, respect, positive)Choi & Pak. Prev Chronic Dis. 2005: 2:1-13

Bias – Question Design

Missing or inadequate data for intended purpose Belief vs. behaviour

• Do you believe smoking is harmful? vs. Do you smoke? Starting time

• In the past year……(changing time reference) Data degradation

• Date of birth vs. age in years vs. age category Insensitive measure

• (worse 1 - 2 - 3 better) limited categories, floor and ceiling effects

Choi & Pak. Prev Chronic Dis. 2005: 2:1-13

Bias – Question Design Faulty scale

Forced choice• Were you happy with the service you received at the diabetes

clinic? Yes No Missing interval

• Complete range of response interval options is not present Overlapping Interval

• Interval anchors or parts of ranges overlap Scale Format

• Odd numbers tend to result in neutral options (choose middle category)

• Even numbers tend to force choice to one side or another• No consensus to best approach


Bias – Question Design Leading questions

Framing• Questions framed so that respondent may choose incorrect answer• What surgery would you prefer?

Outcome with 5% mortality. Outcome with 90% survival.

Leading question• Do you do physical exercise, such as cycling?• Don’t you agree that….• Please rate our excellent service

Mind-set• Try to maintain response option consistency (categories, anchors,

order)


Bias – Question Design Intrusiveness

Reporting Sensitive information

• Both involve selective suppression of personal or confidential information

Inconsistency Case definition

• ICD classifications (first event vs. recurrent)• Coke, soda, pop

Change of scale and/or wording• Scale consistency is especially important if comparisons are made over

time or with the results of other surveyors Diagnostic vogue

• Same illness may have different labels (region, time, type of respondent)Choi & Pak. Prev Chronic Dis. 2005: 2:1-13

5) Pre-test or ‘pilot’ the questionnaire5) Pre-test or ‘pilot’ the questionnaire

Survey Design ProcessSurvey Design Process

1) Define objectives and requirements.1) Define objectives and requirements.

Keep “need to know” questions, be cautious Keep “need to know” questions, be cautious about “like to know”.about “like to know”.

2) Consult with experts familiar with, or are part 2) Consult with experts familiar with, or are part of interest group.of interest group.

3) Draft questions while thinking about data 3) Draft questions while thinking about data collection method and burden on respondent.collection method and burden on respondent.

4) Review/revise the questionnaire.4) Review/revise the questionnaire.

Keys points to rememberKeys points to remember

Write in everyday terms.Write in everyday terms. Follow basic writing principles (direct/to the Follow basic writing principles (direct/to the

point, no spelling errors, grammar etc).point, no spelling errors, grammar etc). Use consistent scales.Use consistent scales. Use consistent wording.Use consistent wording. Be clear about directions (what you would like Be clear about directions (what you would like

the respondent to do).the respondent to do).

Reliability and Validity Reliability - degree to which an instrument

measures the same way each time it is used under the same condition with the same subjects.

Validity - degree to which a survey accurately reflects or assesses the specific concept that the researcher is attempting to measure.

There are many reliable and valid surveys that might be suitable for your research.

Internet Electronic databases – CINAHL, Medline Provincial or National Surveys

Validated surveys are only valid if use entire tool.

Choosing a Ready-Made Survey

Reliability Test-retest Internal Consistency

Validity Predictive - is it correlated with a known outcome? Concurrent - is it correlated with a known and

accepted measure/survey? Content - does it contain all appropriate information

(based on theory and expert opinion)? Construct – does the survey measure what it is

supposed to measure.

Survey LengthSurvey Length

How long is too long?How long is too long?

Avoid long surveys when they Avoid long surveys when they are unnecessary!are unnecessary!

Sequencing and LayoutSequencing and Layout

Intro ParagraphIntro Paragraph Always begin a survey with an introductory Always begin a survey with an introductory

statement.statement.

Often includes:Often includes: Asking the participant to participate.Asking the participant to participate. Ends with thanking the participant for participating.Ends with thanking the participant for participating. Discussing confidentiality.Discussing confidentiality. Purpose of survey.Purpose of survey. Discusses sharing of findings with participants.Discusses sharing of findings with participants.


Begin with easy questions Begin with easy questions (demographics).(demographics).

Group questions by topic.Group questions by topic. Respect chronological order when Respect chronological order when

appropriate.appropriate. Always include comments section at end.Always include comments section at end. No name on survey.No name on survey. Reduce number of “skip to” questions Reduce number of “skip to” questions

(easy for web-based surveys).(easy for web-based surveys).

Use at least 12 pt font (larger for older Use at least 12 pt font (larger for older audience).audience).

PastelPastel colorscolors workwork well.well. Instructions in different style (i.e. Instructions in different style (i.e. ItalicsItalics).).


Copyright ©2008 Canadian Medical Association or its licensors

Burns, K. E.A. et al. CMAJ 2008;179:245-252

Maximizing Survey Response

Data Coding and AnalysesData Coding and Analyses

Once you have decided on survey Once you have decided on survey questions, it is time to think about coding questions, it is time to think about coding the data.the data.

Important to think about Important to think about beforebefore the the collection of data!collection of data!

Coded data can be entered into a Coded data can be entered into a spreadsheet, which will help when spreadsheet, which will help when analyzing data.analyzing data.

Data should be coded numerically for ease Data should be coded numerically for ease of analysis.of analysis.

CodebookCodebook

What is a codebook?What is a codebook? A codebook is a log of your variables A codebook is a log of your variables

(survey items) and how you will code (survey items) and how you will code them.them.

A codebook will help everyone A codebook will help everyone understand the coding schemes to understand the coding schemes to ensure that they are on the same page!ensure that they are on the same page!

Data Processing and Analyses: Data Processing and Analyses: Codebook ExampleCodebook Example

VariableVariable NameName

VariableVariable LabelLabel

ValuesValues CodingCoding MissingMissing VariableVariable TypeType

ageage ageage 1,2,3,4,51,2,3,4,5 1=10-20 years 1=10-20 years 2=21-30 years 2=21-30 years 3=31-40 years 3=31-40 years 4=41-50 years 4=41-50 years 5=51+ years5=51+ years

97=Incorrect 97=Incorrect responseresponse

98=No response98=No response99=Not 99=Not

ApplicableApplicable

OrdinalOrdinal

sexsex sexsex 1,21,2 1=male, 2=female1=male, 2=female 97=Incorrect 97=Incorrect responseresponse



NominalNominal

happinesshappiness happiness happiness atat

workwork

1,2,31,2,3 1=not happy1=not happy2=somewhat happy2=somewhat happy3=very happy3=very happy

97=Incorrect 97=Incorrect responseresponse



OrdinalOrdinal

Spreadsheet ExampleSpreadsheet ExampleID# Age Sex

Happiness 1 1 1 2

2 2 2 2

3 3 1 2

4 57 2 2

5 45 2 3

6 66 2 3

7 2 2 3

8 88 2 3

Open-ended Data codingOpen-ended Data coding

It’s easy to code closed response or rating It’s easy to code closed response or rating questions, but how do you code open-ended questions, but how do you code open-ended data?data?

ObjectiveObjective: to create codes and classify : to create codes and classify responses into categories respondents would responses into categories respondents would have chosen, had they been offered categories.have chosen, had they been offered categories.

Two phases: 1) Scan responses, 2) Scan responses Two phases: 1) Scan responses, 2) Scan responses and then code them.and then code them.

Themes will emerge.Themes will emerge.

Data CleaningData Cleaning

There are several ways you can clean survey There are several ways you can clean survey data.data.

Editing checks:Editing checks:

1) Structure checks-identify non-response.1) Structure checks-identify non-response. 2) Range edits- make sure there are valid 2) Range edits- make sure there are valid

ranges (E.g. No 7’s on a scale of 1-5).ranges (E.g. No 7’s on a scale of 1-5). 3) Make sure ‘not stated’ codes are put into 3) Make sure ‘not stated’ codes are put into

unanswered responses.unanswered responses.

Data Processing and Analysis: Data Processing and Analysis: SURVEY SAYS!SURVEY SAYS!

Response RateResponse Rate

The The actual number of completed surveysactual number of completed surveys, NOT , NOT the number of surveys distributed.the number of surveys distributed.

Low response rates (of less than 60%) may put Low response rates (of less than 60%) may put you at risk for non-response error.you at risk for non-response error. Non-Response error: People who do not respond may Non-Response error: People who do not respond may

be different from those that did, in ways that are be different from those that did, in ways that are important to your study .important to your study .

Try to get the highest response rate possible.Try to get the highest response rate possible.• RemindersReminders• OversampleOversample• IncentivesIncentives

Descriptive or Inferential Statistics?Descriptive or Inferential Statistics?

Descriptive Statistics.Descriptive Statistics. Descriptive statistics are used to describe the Descriptive statistics are used to describe the

basic features of data. basic features of data. Provide simple summaries about the sample and Provide simple summaries about the sample and

the measures. the measures.

Example:Example: Measures of central tendency (means, medians, Measures of central tendency (means, medians,

modes).modes). Frequencies.Frequencies.

Central TendencyCentral Tendency The level of data will dictate which measure of central The level of data will dictate which measure of central

tendency you should use.tendency you should use. CategoricalCategorical = = Data that is classified into categories and Data that is classified into categories and

cannot be arranged in any particular ordercannot be arranged in any particular order (e.g. Apples (e.g. Apples and pears, gender, eye colour, ethnicity). and pears, gender, eye colour, ethnicity).

OrdinalOrdinal = Data ordered, but distance between intervals = Data ordered, but distance between intervals not always equal. (e.g. Low, middle and high income).not always equal. (e.g. Low, middle and high income).

Continuous Continuous == equal distance between each interval equal distance between each interval (e.g. 1,2,3., age).(e.g. 1,2,3., age).

If data is categorical – If data is categorical – MODE.MODE. If data is ordinal – If data is ordinal – MEDIAN.MEDIAN. If data is continuous – If data is continuous – MEAN.MEAN.

Central TendencyCentral Tendency

What is a What is a meanmean?? The sum of all the The sum of all the

scores divided by the scores divided by the number of scores.number of scores.

Often referred to as Often referred to as the average.the average.

Good measure of Good measure of central tendencycentral tendency..

Central tendency is Central tendency is simply the location of simply the location of the middle in a the middle in a distribution of scores.distribution of scores.

MeanMean

Central TendencyCentral Tendency

A A medianmedian is the middle of a distribution. is the middle of a distribution. Half the scores are above the median and half are below Half the scores are above the median and half are below

the median. the median. How do I compute the How do I compute the medianmedian??

• If there is an odd number of numbers, the median If there is an odd number of numbers, the median is the middle number. For example, the median of is the middle number. For example, the median of 5, 8, and 11 is 8.5, 8, and 11 is 8.

• If there is an even number of numbers, the median If there is an even number of numbers, the median is the mean of the two middle numbers. The is the mean of the two middle numbers. The median of the numbers 4, 8, 9, 13 is (8+9)/2 =8.5.median of the numbers 4, 8, 9, 13 is (8+9)/2 =8.5.

Central TendencyCentral TendencyWhat is a What is a modemode?? Most frequently occurring score in a distribution.Most frequently occurring score in a distribution. Distributions can have more than one mode, called Distributions can have more than one mode, called

"multimodal“."multimodal“.

Inferential StatisticsInferential Statistics Use inferential statistics when trying to Use inferential statistics when trying to

reach conclusions that extend beyond the reach conclusions that extend beyond the immediate data alone. immediate data alone.

Examine relationships between variables, Examine relationships between variables, comparisons etc.comparisons etc.

Make conclusions about the population Make conclusions about the population from the sample. from the sample. Requires probability sampling.Requires probability sampling.


Your sampling method and question, or Your sampling method and question, or reason you are conducting the survey will reason you are conducting the survey will dictate the analysis.dictate the analysis.

Descriptive statistics are a common way to Descriptive statistics are a common way to analyze survey data.analyze survey data.


Post-analysisPost-analysis Once you have analyzed your data, it is time Once you have analyzed your data, it is time

to interpret the findings.to interpret the findings.

What do your findings mean?What do your findings mean? Depends on the purpose of the survey.Depends on the purpose of the survey.

• Program evaluationProgram evaluation• ResearchResearch• Customer Satisfaction (gap analysis)Customer Satisfaction (gap analysis)

Are you surprised by the information?Are you surprised by the information? Can you use the information to improve, support or Can you use the information to improve, support or

develop a program?develop a program? Has the information collected raised other Has the information collected raised other

questions?questions?

Documents

1 1 Designing Surveys for Research and Evaluation © Fraser Health Authority, 2007 The Fraser Health Authority (“FH”) authorizes the use, reproduction and/or