Instrumentation: Performance & Surveys

47.269: Research I

Spring 2010

Dr. Leonard

Measuring performanceCan measure optimal or typical performance

Optimal - achievement, aptitude, intelligence testsTypical - everyday behavior, personality, attitudes

Could measure performance through observation, physiological measures, or self-report tests/measures

Self-report is most common in psychology

Surveys and interviews are two different forms of self-report measuresUsed in research that is…

DescriptiveExperimentalNon-experimental/Correlational

Scales of measurementBased on what you are measuring (e.g., behavior, attitudes, emotion, etc.), you may use a different scale of measurement

Degree of specificity in your dataEither discrete or continuous variables

Discrete variables measured as units on scale with no value in between Continuous variables can be any value along a scale to infinity

Scales of measurement:NominalNominal

CategoriesOrdinalOrdinal

Categories that can be rankedInterval Interval

Scores with equidistant intervals between themRatioRatio

Scores with equidistant intervals and absolute zero

Scales of measurementNominal Nominal

OrdinalOrdinal

IntervalInterval

RatioRatio

Self-report designs: Pros

Relatively easy way to collect large amounts of data very quickly

Surveys are cheap and can be self-administered (e.g., online)

Written surveys can be given to a large number of people at the same time and can be anonymous, which may promote honest responses

Interviews are expensive but increase response rate and allow for better understanding of questions

Self-report designs: Cons

• Data are subject to bias, social desirability, demand characteristics, and response sets, which all affect the validity of findings• Bias - researcher’s or participant’s• Social desirability - responding in a way that would be seen

as socially acceptable, especially when the topic is sensitive• Demand characteristics - fatigue, memory burden, confusion• Response sets - straight-line, extremes, right down middle

• Interviews may taint data if participant is trying to impress interviewer or if the interviewer asks questions in a biased way– Option: telephone interviews

Mistakes to avoid with survey questions Mitchell & Jolly (2007)

1. Leading questions

2. Questions that invite social desirability

3. Double-barreled questions

4. Long questions

5. Negations

6. Irrelevant questions

7. Poorly worded response options

8. Big words

9. Ambiguous words and phrases

Planning a survey1. Determine all questions you need

answered, create a list2. Choose the appropriate format for your

questions3. Edit questions for clarity4. Sequence your questions effectively to

avoid order or demand effectsMay help to counterbalance the order!

5. Pilot survey and refine questions6. Choose appropriate sampling strategy

Formatting survey questionsDecide which format will best fit the type of data you want (N, O, I, R)Close-ended

Dichotomous, Yes/NoReliableMay be less powerful

Multiple choice, categoriesCaution not to treat them as interval

Likert scalesOffers choices of response along a spectrumMay be less reliable, ambiguity of points along scaleMore powerful, more sophisticated analyses

Open-endedMay result in numeric or quantifiable dataMay provide more rich, explanatory informationMay not be answered as you intended, coding difficulties

Scales of measurementNominal Nominal

discretediscrete

OrdinalOrdinaldiscretediscrete

IntervalIntervaldiscretediscrete

RatioRatiocontinuouscontinuous

Sequencing survey questions

Overall goal is to increase response rate and accuracy of responsesPut innocuous questions first and personal questions last

Put demographic questions lastKeep similar topic questions togetherKeep questions with similar response options together

You may want to provide response scale only once to save spaceYou may want to use reverse scoring (change direction/meaning) to keep participants engaged and to test whether they are responding consistently

Good example of reverse scoring: Rosenberg Self-Esteem Scale

1= Strongly Disagree, 2, 3, 4= Neutral, 5, 6, 7 = Strongly Agree

_____1. At times I think I am no good at all.*_____2. I feel that I have a number of good qualities. _____3. All in all, I am inclined to think that I am a failure.*_____4. I am able to do things as well as most people. _____5. I feel that I do not have much to be proud of.* _____6. I take a positive attitude towards myself. _____7. On the whole, I am satisfied with myself. _____8. I wish I could have more respect for myself.* _____9. I certainly feel useless at times.*_____10. I feel that I am a person of worth, at least on an equal basis

with others.

*Reverse scored (1, 3, 5, 8, and 9)

Piloting and refining questionsCheck for overall professional appearance of survey and ease of navigating questions

Avoid skips when possibleDon’t let question run across 2 pagesBuild in breaks?

PROOFREAD-mistakes are distractions!Pilot survey

Carefully consider who would provide honest feedback about the questionsTry to create a pilot sample that would represent your target sample

Practice coding responses because that may lead to further refining that makes analysis easier

Minimizing social desirability

Patten recommends 1) observing instead or 2) using projective techniques -Ambiguous task or question participant can’t guess meaning of (e.g., ink blot)

Interpersonal techniques:

Professional conduct; increase respect for research

Create a sense of comfort and security

Reassure participants of confidentiality of responses

Never look at the measure in front of them

Keep multiple participants separated during measurement

Immediately separate any identifiers from data

Minimizing social desirabilityMeasurement techniques:

Place most personal questions mid- to late- in questionnaire

Do not put measures you are relating adjacent to each other

Include some fillers or distracting questions

Include social desirability test items

“I never swear”

“I typically try to help those in need”