1

Surveys, interviews, and diary studies

Michelle Mazurek

(some slides adapted from Blase Ur, Lorrie Cranor, and Rich Shay)

2

Today’s class

•  Logistics: project groups and proposals

•  Surveys

•  Crowdsourcing (Mechanical Turk)

•  Interviews

•  Diary and ESM studies

3

Logistics

•  Project groups

•  Project proposals: https://myelms.umd.edu/courses/1134544/pages/course-project

•  “Talk talk” this afternoon: 2pm CSIC 3117

4

SURVEYS

5

Why a survey?

•  A little bit of data (each) from a lot of people

•  Quantitative results

–  Generalizable if done correctly

•  Quick, easy, unobtrusive, relatively cheap

•  Shallow data

–  Multiple choice, short free-response

•  Biases: self-reported, question/answer order, etc.

6

Survey best practices

•  Pilot, pilot, pilot!–  Ensure questions are neutral, are not ambiguous–  Test different question wordings

•  Consider your sample

•  Include attention checks

•  Don’t make it too long

–  No shortcuts (branch questions equally)

•  Offer option not to answer (avoid lying)

7

Try it!

In groups of 2-3, write a 5-question survey about privacy for student records.

8

CROWDSOURCED STUDIES Mechanical Turk and friends

9

Why crowdsourcing?

•  Many participants, geographically distributed–  More diverse than students *

•  Easy to recruit, screen, assign conditions, pay

•  Most popular: Mechanical Turk

–  Others: Crowdflower, Crowdsource.com, Samasource

10

How it works

Task Requester

Worker Worker Worker

$ $ $✕

http

://up

load

.wik

imed

ia.o

rg/w

ikip

edia

/com

mon

s/4/

40/T

urk-

with

-per

son.

jpg

11

Limitations and risks

•  Can’t observe participants or follow up–  Piloting is especially important

•  Some users enter garbage

–  Collect lots of data–  Pay more than average–  Don’t provide a “shortcut path”–  Use quality checks: trivial, nonsense, repeats

12

Limitations and risks

•  Population: Young, tech-savvy, many from India–  Buhr 2011, Ipeirotis 2010, Ross 2010, others–  Can restrict to American IPs when necessary

•  Measures to prevent repeat participants–  Cookies, IP tracking, MTurk ID list–  Especially if you pay well

•  Turker discussion boards

–  If your study is game-able, will be reported to others

13

Details and procedures

•  Short recruitment text

•  Often, link to external task

–  Built-in features are limited–  Survey in qualtrics (https://umd.az1.qualtrics.com/)–  Custom task site you built–  CMU management infrastructure: SHELF

•  You still need a consent form

–  I have a sample

14

Paying participants

•  When the participant has finished, you notify MTurk and the participant is paid

–  Important for your homeworks!

•  Payment is taken from prepaid MTurk account.

15

Other useful features

•  Screen and reject workers–  Location, quality rating

•  Send notifications (e.g. to come back for part 2)

•  Prevent repeated workers in the same task–  May need multiple tasks per study

•  On average, 100 participants / day

–  Starts faster, slows down, repost

16

Twitter regrets (Sleeper et al.)

•  Mturk survey of 1,221 participants

•  Compared conversational and Twitter regrets

•  Emotional state, awareness, repair strategies

17

Twitter regrets

•  Note the research questions in the introduction

•  Why did they screen for Twitter users age 18+ in the USA?

•  Is conversational regret the right parallel?

•  How was Mturk quality control done?

•  How was the data coded?

•  Limitations

18

INTERVIEWS

19

Why an interview

•  Rich data (from fewer people)

•  Good for exploration

–  When you aren’t sure what you’ll find–  Helps identify themes, gain new perspectives

•  Usually cannot generalize quantitatively

•  Potential for bias (conducting, analyzing)

•  Structured vs. semi-structured

20

Interview best practices

•  Make participants comfortable

•  Avoid leading questions

•  Support whatever participants say–  Don’t make them feel incorrect or stupid

•  Know when to ask a follow-up

•  Get a broad range of participants (hard)

21

DIARY STUDIES

22

Why do a diary study?

•  Rich longitudinal data (from a few participants)–  In the field … ish

•  Natural reactions and occurences

–  Existence and quantity of phenomena–  User reactions in the moment rather than via recall

•  Lots of work for you and your participants

•  On paper vs. technology-mediated

23

Experience sampling

•  Kind of a prompted diary

•  Send participants a stimulus when they are in their natural life, not in the lab

24

Diary / ESM best practices

•  When will an entry be recorded?–  How often? Over what time period?

•  How long will it take to record an entry?

–  How structured is the response?

•  Pay well

–  Pay per response, but don’t create bias

25

Facebook regrets (Wang et al.)

•  Online survey, interviews, diary study, 2nd survey

•  What do people regret posting? Why?

•  How do users mitigate?

26

FB regrets – Interviews

•  Semi-structured, in-person, in-lab

•  Recruiting via Craigslist

–  Why pre-screen questionnaire?–  19/301

•  How were they coded?

27

FB regrets – Diary study

•  12 of 19 participants from the interview participated at least one day

•  Facebook activities, incidents

•  Online form, open-ended questions

–  “Have you changed anything in your privacy settings? What and why?”

–  “Have you posted something on Facebook and then regretted doing it? Why and what happened?”

–  22+ days of entries: $15

28

Location-sharing (Consolvo et al.)

•  Whether and what about location to disclose–  To people you know

•  Preliminary interview

–  Buddy list, expected preferences

•  Two-week ESM (simulated location requests)

•  Final interview to reflect on experience

29

ESM study

•  Whether to disclose or not, and why–  Customized askers, customized context questions–  If so, how granular?–  Where are you and what are you doing?–  One-time or standing request

•  $60-$250 to maximize participation

•  Average response rate: above 90%