Survey Design: Designing

Questionnaires

Ronda L. Cochran, MPHOffice of Prevention Research and

EvaluationDivision of Healthcare Quality Promotion

Centers for Disease Control and Prevention

Why Survey? - To know or not to know… that is

the question To find out the characteristics,

behavior, or opinions of a particular population (i.e., healthcare personnel in long term care centers)

To measure the degree to which an individual exhibits the characteristic(s) of interest (i.e., use of alcohol-based handrub)

A Simple Example Do you eat candy?

How often do you eat candy?

□Yes □No

□Never…□Rarely…□Sometimes…□Often…□All the time

Survey Research Most frequent mode of

observation/measurement Exploratory, descriptive, and

explanatory Unit of analysis usually individuals

(respondents) Used most often for:

describing a population too large to observe (practices/behaviors) directly

measuring attitudes Common terms: Survey, Poll,

Evaluation, Assessment, Questionnaire…

Modes of Survey Administration

Face-to-Face

Computer-assisted

Telephone

Interactive Voice Response

Surveys

Mail

Internet

Types of Survey Research

Self Administered Questionnaires: Respondents complete the

questionnaire themselves (e.g., mail survey)

Interview: researchers ask the questions and

record respondents answers (e.g., face-to-face interview and telephone interviews)

Self-Administered Questionnaire

Mail Survey: Cover letter of explanation (why

is this important to complete???) Self-addressed stamped return

envelope

But… Why not return?

Self-Administered Questionnaire

Response Rate: how many respondedhow many mailed

Representativeness of the sample

Higher response rate, less response bias

Depends on target audience How to increase???

Follow-up mailings Non respondents All respondents

Interview Surveys

Usually have higher response rate than mail surveys

Less “do not knows” or blanks Less items misunderstood Additional observations possible

(e.g., ability to speak/understand English, general reactions)

Interview Surveys

Guidelines for Interviewing Appearance Attitude/Demeanor Familiar with survey Follow wording exactly Record responses exactly Probe (request elaboration)

appropriately Trained and supervised

Survey Research and Technology

CATI - computer assisted telephone

interviewing CAPI - computer assisted personal

interviewing

CASI - computer assisted self-interviewing

CSAQ - computerized self-administered questionnaire

TDE - touchtone data entry

VR - voice recognition

Define the purpose

Write rules (e.g., for administration)

Determine survey format

Write itemsIdentify

respondents

Write objectives

General Steps in Survey Development

Constructing a Survey First & foremost, decide what

information you need and why you need it Define the purpose of the survey –

be specific (i.e., assess extent to which occupational exposure management policies and practices are available and consistent

with CDC guidelines)

Think in terms of results – prepare hypothetical tables – what you put in… is what you get out…

Constructing a Survey: Determine Data to be

Collected Demographics (e.g., age, gender, years in practice)Consider the respondents:

Who are they?What are their expected levels of

language and reading skills?How familiar/aware are they with

the topic ?How will they be recruited?How much time will they have to

complete the survey?

Constructing a Survey: Determine Data to be

Collected Variables important to testing

hypothesis/research questions (e.g., independent variables, outcomes, time in study [if applicable]) – date of admission/discharge)

Research questions/objectives should guide the selection of items

Write theoretical definition

Write items

Identify formatIdentify

sources of items

Determinenumber of items

Createblueprint

Identify conceptdimensions

Steps in Item Writing

Item Sources….Generating

Items Experts

Literature (review – can often find templates for item phrasing)

Interviews (information from members of the targeted population)

Other Instruments (designed to measure the concept/area of study)

Types of Questions Open-ended: respondent

provides his or her own answer to the question What is the most common problem

associated with survey construction? Close-ended: respondent is asked

to select an answer from among a list provided (response choices) The most common problem

associated with survey construction is:A. Inadequate instructionsB. Use of vague or unfamiliar termsC. Overall format of the instrument D. All of the above

Responses Response choices/categories

should be EXHAUSTIVE Other (Please

specify):________________ Response choices/categories

should be MUTUALLY EXCLUSIVE How many times have you visited the

doctor over the past 6 months?A. 0-1 A. 0-1B. 1-2 vs. B. 2-3C. 2 or more C. 4 or more

Question Development Writing Good Questions

Clear and unambiguous Spell out acronyms and define unusual

terms Use simple words (reading level) Give specific time frame/period (for recall) –

i.e., “past 12 months”

Relevant to the respondent (Do not know, no opinion, undecided, not applicable… however, use selectively)

Short items are best

Question Development Writing Bad Questions

Avoid double-barreled questions (questions with multiple parts)

Patient safety applies to patients and healthcare personnel.

Patient safety initiatives should be funded on a national level without any tort reform.

Avoid negative items (negation leads to misinterpretation)

Patient safety should not be a national priority.

Avoid biased items and terms (noting social desirability of questions and answers)

Questionnaire Construction

Format - spread out, uncluttered, not squeezed on to one page, response choices should be clear

Contingency questions: survey question intended for only some respondents, determined by their responses to some other question (skip patterns) – i.e., “Does your facility provide Hepatitis B vaccine to healthcare personnel? If NO, Go to #10” (skipping questions on Hepatitis B vaccine administration practices)

Questionnaire Construction

Order and group items and answers to use space efficiently and effectively Be careful of -- Response sets:

respondents develop a pattern of responses (e.g., agreeing with all statements) -- Alternate statements/questions

INSTRUCTIONS!!! (and transitions too)

Pretesting the questionnaire (pilot): writing rules for administration and scoring

Survey Data Entry and Analysis

Clear measurement rules for administration and scoring:Standardized - process applied the same way each time (e.g., duplicate data entry/verification)Explicit - all users able to follow the same process Unambiguous - clear rules that cover all possibilities (e.g., missing data)Subject to evaluation - modify based on experience

Measurement Rules

For example… Survey Construction – one must consider: How a correct response is scored? How incorrect response is scored? What to do about skips? What to do with >1 answer to an

item?

Cognitive design principles & formatting

Simple design – reduce noise, include only the information you need

Use alternatives to grid lines – dot leaders and response boxes De-emphasize irrelevant info – use footers for copyright and

other related info Use response boxes Emphasize selected terms – use judiciously and consider

underlining Avoid repeated instructions Use a consistent design throughout the questionnaire Order scale options consistently – use negative and positive

item wording to control acquiescence Place instructions in the context of the questions Group concepts Create a natural reading flow (left to right) Positioning question screens and skip instructions – place to

the right at the appropriate location, consider arrows Use left justification Use graphical guides to enhance tracking (upside down

Triangle) Use spacing to help respondent distinguish between sections Enhance interest with a title page – may include graphics

What you put in…is what you get out…

Be specific about what new information you need and why

Choose the survey method that works best for you and the situation

Determine sampling method(s) Write good questions that will provide useful,

accurate information Design and test an instrument that is easy

and interesting to answer Put together the necessary mix of people,

equipment, and supplies in the necessary time frame (logistics)

Analyze the data Share and present your results

Find the problem…

Do you exercise regularly?□ Yes □ NoVague wording (regularly); what is considered exercise?

Find the problem…

How many times have you been admitted to a hospital?_____Reference period not included

Find the problem…

Do you exercise and eat a low-fat diet? □ Yes □ NoDouble-barreled; what is low-fat?

Find the problem…

What percentage of your weekly grocery bill is spent on dairy products?_____ PERCENTRequires a calculation, what counts as dairy? Consider listing items and asking for a total.

‘The ability to ask the right question is more than half the battle of finding the answer.’ -Thomas J. Watson (1874– 1956, president of IBM)

*The finding and conclusions in this presentation are those of the author and do not necessarilyrepresent the official position of the Centers for Disease Control and Prevention.