Survey design basics

ULS Leadership ProgrammeBerenika M. Webster, PhD

November 2014

Learning outcomes

By the end of this session participants will acquire understanding of the following aspect of survey design process• What be measured with a survey?• Roles (sponsor vs. researcher)• Survey process

• Design (formulating research question)• Question development• Testing (and train)• Data collection• Analysis

Reasons for conducting surveys

• To uncover the answers. Learn about what motivates survey respondents and what is important to them, gather meaningful opinions, comments, and feedback.

• To evoke discussion. Give your survey respondents an opportunity to discuss important key topics. Communicate with your respondents about your survey topic. This allows you to dig deeper into your survey, and can incite topics related to your survey within a broader perspective.

• To provide objective evidence for decisions. Conducting surveys is an unbiased approach to decision-making. Don’t rely on “gut feelings” to make important business decisions. You can collect unbiased survey data and develop sensible decisions based on analysed results. By analysing results, you can immediately address topics of importance, rather than waste time and resources on areas of little concern.

• Compare results. Survey results provide a snapshot of the attitudes and behaviours of your target populations. This feedback is your baseline to measure and establish a benchmark from which to compare results over time.

When NOT to conduct surveys…

• Your reason for doing the survey starts with, “It would be nice to know….”

• You don’t know much about your target audience and you want to use the survey to learn about them (focus groups or personal interviews would be a better first step).

• You have already made your decision; you just want to see how it is going to be received.

• You don’t have the enough resources to act on the information you get back.

• You don’t have enough money to do your survey right without cutting corners.

• You’ve never done a survey before and you aren’t getting help from someone who has.

Before you design your survey…

• You should be clear about…• The main goal of the survey• How you going to apply the survey data• The decisions you will make as a result of the

survey data• Who are your target population(s)

• Write outline of your final report…

Measuring awareness

• Awareness questions measure consciousness of particular issues.

• Asking how familiar people are or how much people think they know is acceptable when measuring public awareness.

• An example is, “Before this survey, were you very, somewhat, or not at all familiar with the Liaison Librarian services at ULS?”

Measuring opinions• Opinion questions measure beliefs for ideas held with

confidence but not substantiated by direct proof or knowledge.

Examples: “In general, how effective would you say freshman composition library instruction is in preparing students for their research assignments? Would you say they are very effective, somewhat effective, or not at all effective? “In the past three years, would you say the WIFI performance in Hillman has improved, stayed the same, or gotten worse?”

Measuring perceptions• Measures the distance (gap) between the ideal and

the image(perception) profile of the respondent• Can use semantic differential or distance scalesMy pizza was: hot _ _ _ _ _ _ _ _ cold

fresh_ _ _ _ _ _ _ _ stale

My job can be described as 1(Not at all) – 7 (Perfectly)

__ Boring __Demanding

__ Easy __ Satisfying

__Rewarding __ Degrading

Measuring behaviours• Behavior and activity questions inquire about

what people actually do: What? Where? When? and how often?

My day at Hillman

Adding context by measuring• lifestyle patters• demographics• affiliations

Measuring needs• Involves identification of relevant needs and asking respondents

to rank them (e.g. forced comparison or paired comparison)Rank in order of importance to you (from1 to 3)

More outlets ___

Better WIFI ____

Better lighting ___

For each pair of choices indicate that which is more important to you

More outlets ______ Better lighting ____ Better WIFI _____

Better lighting ______ Better WIFI ______ More outlets ______

Measuring attitudes• Attitude questions measure states of mind or feelings. • Attitudes cannot be measured with just one question

Generally, a battery of questions is needed to identify and assess attitudes.

• Attitudes measurement = knowledge, feeling and action• Knowledge: Are you aware? Have you heard? List or name

….• Feelings: scale to show direction and distance from neutral• Action: ask about past, present of future behaviour or create

a hypothetical scenario

Measuring decisions

• …or the process of decision making• Information sources and evaluation

criteria are two components of decision making process that can be measured through survey

• Sources of information: personal knowledge, social influence and media sources

• Evaluation criteria: respondents may rank attributes in order of importance

Roles: Survey sponsor

• Explains WHAT information is needed, WHY it is needed and HOW it may be used (frankly!)

Alreck and Settle, 3rd ed. 2000

Roles: Researcher

• Advises sponsor on capabilities and limitations of research tools

• Understands sponsor’s questions, motivations and intentions

Survey design 1• What do I want to measure in my survey?

• Attitudes, perceptions, needs, behaviours

• What will I ask to measure it?• What are characteristics of the variable I want to measure and what

questions should I use to directly/indirectly measure that variable• Are there any valid pre-existing scales/questions I could use?

• How will I measure it?• Dichotomous scale (yes/no; true/false), • Nominal scale (allow for several choices that are not ordered),• Ordinal scale (can be ordered on continuum), • Rank ordered (from most xxx to least xxx)

• Interval scale (allows for calculating mean) • Numeric or semantic Likert-type scale• Continuous (answers placed anywhere on a scale)

Survey design 2• How will I check for validity?

• Valid means that survey questions measure what they are intended to measure (and not some other concept)

• Confirm with people knowledgeable in the field and pilot

• How will I administer the test?• Timing (work backwards from the date results are needed by; consider best

times to contact respondents; need approvals, consider workloads)• Mode (how do I contact respondents; email; paper; f-2-f or phone interview)

• How will I check for reliability?• Reliable survey questions measure consistently and accurately what is was

designed to measure

• What will I do with the results?• What is “good” or “bad” result?• How will I use this to make decisions?• What are the results NOT telling me?• Have outline of final report prepared BEFORE you start designing your

survey

Testing for reliability

• Test and re-test: administer test to the same group multiple times and check for consistency of responses

• Conduct factor analysis: see if there are correlations in answers to similar questions (e.g. Cronbach coefficient alpha when want to test reliability of Likert-type scales)

Open-ended questions• Allow respondents to write answers “in their own

words”• Potential for rich source of information and can

reveal what respondents value/care for the mostBut• Need coding and scoring by multiple “experts”• High levels of agreement on the above shows validity

• Greater risk of respondents misunderstanding the question

Closed-ended questions

• Easy in respondents – no need to articulate answers• Less potential for ambiguity (in how responded interprets

question and how responses are analysed)• Easy to classify (code) making analysis straightforwardBut• “Suggests” answers that respondents did not consider

before• Gives respondents no opportunity to give a response

different from that listedAnother option: “open response” question

Open response questions

• Eliminates disadvantages of closed and open questions

• Allows for both specific and open responses

Attributes of a good survey question

• Focusing directly on a single, specific issue• Which brand do you like the best?• Which of these brands are you most likely to buy?

• Brevity – long questions risk of lacking focus or clarity • Can you tell me how many children you have, are they boys or girls and

how old are they?• What is the age and sex of each of your children?

• Clarity – everyone must interpret the question in the same way• How much storage space in your house do you and your

spouse use?• What proportion of the storage space in your house is used

for your things and how much for your spouse’s?

Putting survey questions in a meaningful order and format• Opening questions need to be easy to answer and be

non-threatening (“hook them”)• Follow a logical order (e.g. questions on one topic

grouped together)• Use a variety of question types (to keep in interesting)

but do so in moderation (do not confuse with different scales or types of required response)

• Include sensitive questions towards the end (so you don’t lose them early on); do not save important questions to the end of a long survey (respondents will be bored by them and will want to finish asap)