Diary Studies in HCI & Psychology

0

Diary Studies in HCI & Psychology:

Why They’re Useful and How to

Conduct Them

Demetrios Karis

[email protected]

UPA Boston’s Tenth Annual

Mini UPA Conference

May 25, 2011

1 • © 2011 Karis • Diary Studies •

What is a Diary Study?

In a diary study, participants go about their

normal lives except that they report what they’ve

done or experienced (via questionnaire or freeform

using an electronic or paper diary) at some interval,

signal, or event, and they do this multiple times over

days or weeks.


I’ll Go Over

• Example of what to report – Study on information needs while mobile: ―What time does the post

office close?‖

– Questions:

1. Where were you?

2. What were you doing?

3. What was your information need?

4. I addressed the need (At the time, Later, Not at all)

5. If you attempted to address the need, how did you do so? If

you didn’t make an attempt, why didn’t you?

6. Could you have addressed your need by looking at your

personal data (e.g., email, calendar, web browsing history, chat

history, or other)

• How people report

• When they report

• What they report


Outline

• Why diary studies?

• History

• Example 1: Understanding the mobile phone music experience

• Methodology – Three types of diary designs

– Paper and pencil versus electronic diaries

– The role of media, and the ―snippet‖ technique

– Practical considerations: number of participants, duration, data

collection, and analysis

• Example 2: Information needs while mobile

• Example 3: Electronic chronic pain diary

• Example 4: A diary study of work-related reading

• Related methods – Experience sampling method (ESM)

– Day-in-the-life observation

• References


Why Diary Studies?

• Diary studies (in HCI):

– Provide information on how people are using a device or

application in natural everyday situations – and provide information

on how this use unfolds over time

• When, how, and where they use it

• Problems, similar products they’re using, unmet needs

– Provide information on user needs in order to provide

requirements & design guidance for new devices or applications

– Eliminate biases introduced by retrospection

– Often more accurate than retrospective techniques

• Limitations

– Participants’ commitment and dedication required

– Training sessions often required


History

• Paper and pencil diary studies began appearing in the psychological

literature 70 years ago in the early 1940s

• In psychology, used to study – Personality processes

– Marital and family interaction

– Physical symptoms

– Mental health

• Example: An investigation of women’s perception of intimacy in their

romantic relationships in everyday life, by assessing – The degree of intimacy the typical woman in a committed relationship

feels on average

– The extent to which the typical woman’s feelings of intimacy vary over

time

– Whether women differ from one another in their average feelings of

intimacy and in the variability of their feelings of intimacy over time

This example, and much of this presentation, comes from Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life as

it is Lived. Annual Review of Psychology, 2003, Volume 54, 579-616.

For a review of diary studies in HCI and CSCW prior to 2002, see Palen, L., & Salzman, M. Voice-Mail Diary Studies for

Naturalistic Data Capture under Mobile Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACM

Press, 87 – 95.


Example 1: A Diary Study to Help

Understand the Cellphone Music Experience

Goals Understand how using a phone to listen to music fits into a person’s

―musical ecology‖ -- i.e., how do people use their phone and other

music devices in their daily life?

– More Specific Goals:

• Understand the user experience on a particular phone – with

respect to music – and whether it’s an improvement over the

last version

• Recommend changes to improve future music phones

• Five Studies 1. Expert review

2. Usability study

3. Diary Study

4. Interviews with iPod users

5. User group study of Rhapsody and online music


Methodology

• Week 1: use your phone as you normally would

– Email sent every day asking if they used their phone, and how

• Week 2: a task given each day (e.g., rewind song a few seconds)

– We receive information on usability and utility, and also whether they

had performed the task before

• Interviews at the beginning and end of the study

Kate Dobroth ran this study.


Methodology: More Details

• Nineteen participants between 18-44 (most between 18 and 24) who

used their phone for listening to music (recruited from a list provided

by a wireless carrier)

• Initial interview

– Impressions of the phone, music consumption, and experiences with

loading music onto the phone

• Daily emails about how phone was used

– Did you use your phone today? What did you use it for?

– Did you use any of the music features? If so, which ones?

• Second week of emails included music-related tasks

– Allowed us to explore whether participants had discovered various

features, and whether they were useful

• Final wrap-up interview

– Explore issues and questions from emails


Lots of Questions

• Where are you when you listen to music?

• How do you listen to music? (devices)

• Why do you listen using each type of device?

• How many songs do you have on your phone?

• Getting music onto the phone

– Where do people get music?

– How do they get it onto the phone?

• Usage

– Which phone features are used most often?

– How often do people use the music feature?

• Task performance

– Performance and impressions of specific tasks completed during the second week

• Overall impressions of the phone

– Like most and least


Questions to Consider

• Are these results representative of all people who might like to listen

to music on their phones?

• Would you make design decisions or change requirements based on

these results?

• What other studies might be worth doing?

• Could there be memory issues with a diary study like this? That is, a

person may tell you what they did at the end of the day, many hours

after they actually did it.

• How can diary and quantitative techniques be combined?

• Other questions?


Log Analysis Plus a Diary Study

• In the days before Smartphone apps, users could download a version of

SuperPages onto their feature phones (a BREW application)

• Several usability studies — valuable, but little information on utility

• Combining a diary study and log analysis helped to answer a number of

questions: – What prevented people from using the service, or finding what they wanted?

– Where were they when they used the service, and what was the context?

– What are the opportunities for improvement, to make the service fit in with

how people actually use the application?

– When did people use the service?

– What types of searches were most popular?

– How many searches did people conduct in each session?

– Which location options were used most often?

– Which businesses did people look for?

Adding quantitative data from log analyses helped

tremendously to answer basic questions.

D

L

Both


Three Types of Diary Designs

• Participants can be asked to report on their actions or experiences in

three ways 1. At specific intervals

• Once a day, or specific times of day (after breakfast, after school,

before bed) or specific time intervals (every three waking hours)

2. At a signal from the researcher

• Participants prompted to provide diary reports at fixed, random, or

a combination of fixed and random intervals

3. When an event occurs

• Participants provide reports whenever an event occurs that meets

some pre-established criteria

• How often should participants report? – Need to consider the temporal patterns of the phenomenon being

studied; what’s the expected frequency of the events of interest?

– Also, need to consider the type of event or phenomena

• Pain: need to report immediate experience, not past

• Major events can be reported daily with much less error or bias (did

you take mass transit today?)


Paper and Pencil Diaries

• Easy to implement, but several problems – Participants may fail to remember the scheduled response times

– Or may fail to have diaries at hand

– Potential for retrospection errors: participants rely on reconstruction to

complete missed entries

– For researcher, the burden of data entry and handling

• Suggestions – Make diaries easily portable

– Reduce possibility of participant errors by preprinting the dates and

times of expected responses onto diary sheets

– Reduce demand characteristics by emphasizing the importance of

accuracy over the number of responses

– Pilot test your diary material on a representative sample of people

• Augmented paper diaries – Responses still collected via paper, but these are augmented by

signaling devices such as pagers, preprogrammed wristwatches, phone

calls, or mobile phones (alarms, email, text messages, etc.)


Advantages of Electronic Devices

• Studies started appearing about 2000

• Many advantages – Allow for signaling; can remind you to complete diary entry

– Can date and time-stamp entries

• Prevents ―back-filling‖ diaries to appear compliant

– Can provide location information if device has GPS sensor

– Flexibility in presentation of questions, plus the ability to force

responses before proceeding

– Improved data quality and management, & eliminates data entry

• Data doesn’t need to be transcribed and checked

• Out-of-range responses prevented

• Fewer missing responses – Eliminates risk of skipped questions

– Much higher compliance rates using electronic diaries compared to

paper (in both children and adults)

• More diary entries completed (6.89 days out of 7 for electronic

diaries versus 4.97 using paper in Lewandowski et al., 2009)

• Early studies: participants provided with PDAs; now, many people

carry smartphones with them at all times, and these can be used


Diary Studies: The Role of Media

• Up until now we’ve been assuming that people are just responding to

questions by writing their answers or selecting from multiple choices

• Photos, video, audio recordings, & objects can help in diary studies – Participants capture events (e.g., via a photo); these can stand alone or

can be used to elicit more detail during a later interview

• Annotation produced at the same time can be very beneficial (but

can add significantly to the effort involved)

– Audio can be used in some situations when taking a picture isn’t

appropriate (recording an audio memo or calling a number and

responding to questions)

– In some situations, it may make sense to have participants collect

objects

Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary

Studies. Proceedings of the CHI ‘05 (Portland, OR, April, 2005), ACM Press, 899-908.


The Snippet Technique

• While out and mobile there may not be time to complete a thorough

diary entry

• The snippet technique: ―The in situ capture of snippets: bits of text,

audio, or pictures captured in a matter of seconds…then, at a

convenient time, participants access a website to review their

snippets and complete thorough, structured diary entries.‖ – Use existing mobile device and send text (SMS), pictures (MMS),

voicemail, or a combination

• Preliminary results – Participants wrote more using snippet technique

– Snippet technique doesn’t seem to decrease accuracy or completeness

– Participants typically had a strong preference for one type of media;

some used two, no one used all three

• Text used most, then audio, then pictures (but some had

inadequate cameras)

Brandt, J., Weiss, N., & Klemmer, S. R. (2007). Txt4l8r: Lowering the Burden for Diary Studies Under

Mobile Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308.


Practical Considerations

• Number of participants

– 10 to 20 is typical within CHI

– But wide range: 5 to > 50 • BBC diary study on audio listening: 2,000 respondents

• Duration

– One or two weeks typical

– A day to a month or more

• Reducing drop-out rates and improving data quality

– Recruit ―appropriate‖ participants, interview them if possible, explain

requirements

– Withhold all or most pay until the study is completed

– Keep the time requirements for each diary entry short

– Check to make sure responses written when requested, not later

– Provide examples so participants know what’s expected

– Communicate with participants during study • Personal contact is often more important than monetary incentives in

motivating participants

• Provide encouragement and feedback on responses (is it detailed enough?)

Exact values depend on the situation and study goals.


Techniques for Collecting Data

• Paper diaries

– Still a good idea in some situations (e.g., seniors without computers)

• Email

– Have participants email you their diary entries every day

– We’ve found it effective to email a reminder, which can also include questions or

rating scales

• Mobile phone-based applications/text messages

– Participants text their diary entry, or use a smartphone application

– ―Snippets‖ (SMS, MMS, voicemail) during day as aid for later recall (see Brandt

et al. 2007)

• Some now use Twitter as a snippet technique

• Voicemail

– Participant calls dedicated voicemail line rather than recording event

electronically or on paper

– Free-form or structured (can be IVR system)

– See Palen & Salzman (2002)


Techniques for Collecting Data (2)

• Excel spreadsheet (e.g., worksheets for each day of the week)

• Web-based techniques

– Google docs or spreadsheets & Google forms (https://docs.google.com)

• Free text or preset forms

– Blogs (e.g., http://wordpress.org/)

• One for each participant; can be password protected

– Design a diary as a forum (http://www.phpbb.com/)

• Used by Lichtner et al. (2009)

– Online surveys

https://docs.google.com/

http://wordpress.org/

http://www.phpbb.com/


So you want to run a diary study…

• What are you studying, and what’s the expected frequency? – Do you need to collect information multiple times a day, or is once

sufficient?

– How long should you continue the study?

• How many reports of experiences or device use do you need from

each participant?

• Is reporting event driven? – If not, is reporting at a set interval sufficient, or should you signal the

participant to respond?

• How will people report on their activities or experiences? – Will they be at a desk or mobile?

– Will computers or smartphones be available?

– Will they have web access?

– Will a snippet technique be useful? (e.g., photo or text message during

day, details filled in at night)


Analysis

• In the applied CHI world, the studies are usually qualitative, and

simple descriptive statistics are typically sufficient

– Exceptions, of course: e.g., Czerwinski et al. (2004)

• 10 participants, analysis of task switching, difficulty, length, etc.

• Multivariate logistical regression with each user’s task switch entry

as input

• In academic psychology, multilevel models modified to handle

repeated-measures data are recommended (these are also called

hierarchical linear models)

– Lots of data are often collected; one study on emotion included 2,300

person-days of data


Example 2: Typical Data in HCI

• Keep a diary for two weeks of all your information needs when you’re

mobile. Snippet technique used: Text messages sent to email

address, which were then posted automatically to the web

Later, six questions about each snippet

answered on the web

1. Where were you?

2. What were you doing?

3. What was your information need?

4. I addressed the need (At the time,

Later, Not at all)

5. If you attempted to address the need,

how did you do so? If you didn’t

make an attempt, why didn’t you?

6. Could you have addressed your need

by looking at your personal data (e.g.,

email, calendar, web browsing history,

chat history, or other)

Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. A

Diary Study of Mobile Information Needs. In

Proceedings of CHI ’08 (Florence, Italy, April 2008),

ACM Press, 433-442.


Breakdown of Information Needs

Need

Category

Example % of Total

Diary

Entries

# of

Category

Entries

# of Part.

Reporting

Cat.

Trivia ―What did Bob Marley die of, and

when?‖

18.5% 78 17

Directions ―Directions to Sammy’s Pizza‖ 13.3% 56 17

Point of Interest ―Where is the nearest library or

bookstore?‖

12.4% 52 17

Friend Info ―Where are Sam and Trevor?‖ 7.6% 32 8

Shopping ―How much does the Pantech phone

cost on the AT&T website?‖

7.1% 30 16

Business Hours ―What time does the post office

close?‖

6.9% 29 15

Personal Item ―What is my insurance coverage for

cat scans?‖

6.4% 27 12

First 7 of 16 categories from Table 1 of Sohn et al., 2008.


Example 3: Electronic Chronic Pain Diary

Study to record pain by adolescents with juvenile idiopathic arthritis using a

Palm Tungsten W PDA (2004)

Stinson et al. e-Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents with

Arthritis. The Clinical Journal of Pain, 2006, Volume 22 (3).


Example 4: Requirements for a New Device

It’s 1997, and you want to gather information so you can create a device for

work-related reading (remember: Sony Reader, 2006; Kindle, 2007) – What kinds of reading do you need to support?

– How do people read as part of their jobs?

– What are the implications for design?

• There’s not much literature on this…what should you do?

• One solution: a diary study – Pilot study, to create initial taxonomy of reading activities

– 15 participants, diverse occupations (pilot, surgeon, architect, optician)

– 5 days

– Logged daily document activities during work day

• Including: books, journals, electronic documents, PDAs, post-it

notes

• Used paper log forms, included estimates of duration

– Daily interviews with each participant (30 minutes to 3 hours)

• Interviewer expanded on description: type of document,

collaborative or not, where occurred, etc.


Work-Related Reading: Results

• Taxonomy of reading activities, 10 categories, top three: – Skimming (subcategories: sorting, proofing, browsing)

– Reading to search/answer questions

– Reading for cross-referencing (to integrate information)

• Writing categories – Creation

– Note-taking

– Annotation

– Form-filling

– Updating (calendars/schedules)

• Reading was accompanied by writing most of the time

• Great deal of reading and writing using multiple documents in

parallel

• About half the time participants used at least two display surfaces

concurrently

• Shared document viewing common – Different clusters, with high sharers viewing documents with others over

50% of the time


Design of Digital Reading Devices

―The linear, continuous reading of single documents by people on their

own is an unrealistic characterisation of how people read in the course

of their daily work.‖

• ―Reading occurs more frequently in conjunction with writing than it

does in isolation….[reading devices should] support the marking or

writing of documents during the reading process.‖

• There is a ―need to consider how single display devices can support

the range of cross-document activities people carry out.‖

• Flexible search and navigation is important for some common

activities.

• Collaborative viewing needs to be supported.

Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading:

Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April

1998), ACM Press, 241-248.

Do the Kindle and Nook meet work-related needs?


Four Examples

• Example 1: Understanding the cellphone music experience

– The use of a device (mobile phone) for a specific purpose (listening to

music)

• Example 2: Information needs while mobile

– How existing devices and strategies are used to address a need

• Example 3: Electronic chronic pain diary

– Recording an experience (pain) over time

• Example 4: A diary study of work-related reading

– Understanding the requirements for a new device


Two Related Methods

• Experience Sampling Method (ESM)

– Used to obtain a random selection of everything a person does and

experiences in everyday life

– Can be considered a type of diary study

• Day-in-the-Life Observation

– Observation can provide a more accurate record than diary studies (and

diary studies are often more accurate than surveys)

– But very labor intensive


Experience Sampling Method (ESM)

• Used to obtain a random selection of everything a person does and experiences in

everyday life – Participant signaled 8 or more times a day for 1-2 weeks; then questions/ratings presented

• Used in psychology for almost 40 years to understand the dynamics of mental health,

what and how often people think about things, happiness, moods, etc.

• An excellent method for determining

1. ―What people do all day, where, and with whom‖

2. ―How people report experiences, different moments in their lives, along a great

variety of dimensions‖

―ESM is a means for collecting information about both the context and

content of the daily life of individuals…the unique advantage of ESM

is its ability to capture daily life as it is directly perceived from one

moment to the next, affording an opportunity to examine fluctuations

in the stream of consciousness and the links between the external

context and the contents of the mind.‖

Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method:

Measuring the Quality of Everyday Life. Sage Publications, Inc, 2006


Day-in-the-Life Observation

• Observation can provide a more accurate record than surveys or

diary studies

– One study found that ―next day telephone interviews understate TV use

by 62% and magazine use by 43% when compared with observation;

diaries also understate these media consumption (13% and 29%,

respectively)‖ (Middletown Media Study 1, Ball State University)

• A Day in the Life: An Ethnographic Study of Media Consumption,

Middletown Media Study II, Online Publishers Association, July

2006. (Ball State University, Indiana)

– 350 adults in Muncie and Indianapolis IN were observed in Spring 2005

– Participants were observed on average nearly 13 hours – about 80% of

the waking day (over 5,000 hours in total)

– Every 15 seconds, their media consumption and life activities were

recorded (15 media and 17 life activities tracked; e.g., eating, child care)

– Observations were made at home, work, auto and other locations

• Example on next page


(He works from home.)

(No print usage all day.)


What is a Diary Study?

In a diary study, participants go about their

normal lives except that they report what they’ve

done or experienced (via questionnaire or freeform

using an electronic or paper diary) at some interval,

signal, or event, and they do this multiple times over

days or weeks.

• Participants go about their normal lives

• They report on some activity or experience – Electronic

– Paper

• They report according to some schedule – Interval

– Signal

– Event

• They report repeatedly over days or weeks

• A qualitative technique – Can be combined with surveys, log

analyses or other quantitative

techniques

• Media can be used – Photos, video, audio recordings

• Snippets can be recorded and

expanded upon later

• Practical considerations – Number, duration, reducing drop-out

rates

– Ten techniques for collecting data


References

• Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading:

Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April 1998),

ACM Press, 241-248.

• Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life as it is Lived. Annual Review of

Psychology, 2003, Volume 54, 579-616.

• Brandt, J., Weiss, N., & Klemmer, S. R. Txt4l8r: Lowering the Burden for Diary Studies Under Mobile

Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308.

• Brown, B. A. T., Sellen, A. J., & O’Hara, K. P. A Diary Study of Information Capture in Working Life.

Proceedings of CHI ‘2000 (The Hague, Amsterdam, April 2000). ACM Press, 438 – 445.

• Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary Studies.

Proceedings of CHI ‘05 (Portland, OR, April 2005), ACM Press, 899-908.

• Conducting Commercial Ethnography, GreenBook White Paper: http://www.greenbook.org/marketing-

research.cfm/conducting-commercial-ethnography [This paper includes much practical information].

• Czerwinski, M., Horvitz, E. & White, S. A Diary Study of Task Switching and Interruptions. Proceedings

of CHI ‘04 (Vienna, Austria, April 2004), ACM Press, 175-182.

• Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method: Measuring

the Quality of Everyday Life. Sage Publications, Inc, 2006.

http://www.greenbook.org/marketing-research.cfm/conducting-commercial-ethnography








References (2)

• Lewandowski, A.S., Palermo, T.M., Kirchner, H.L, & Drotar, D. Comparing Diary and Retrospective

Reports of Pain and Activity Restriction in Children and Adolescents with Chronic Pain Conditions. The

Clinical Journal of Pain, 2009, Volume 25 (4), 299 – 306.

• Lichtner, V., Kounkou, A. P., Dotan, A., Kooken, J. P., Maiden, N. A. M. An Online Forum as a User

Diary for Remote Workplace Evaluation of a Work-Integrated Learning System. Proceedings of CHI

’09 (Boston, MA, April 2009), ACM Press, 2955-2969.

• Palen, L., & Salzman, M. Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile

Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACM Press, 87 – 95.

• Papper, R., Holmes, M., & Popovich, M. (2004). Middletown Media Studies 1. Center for Media

Design, Ball State University. Available free at

http://cms.bsu.edu/Academics/CentersandInstitutes/CMD/InsightandResearch/Capabilities/ProjectGall

ery/MiddletownMediaStudies/MiddletownMediaStudiesI.aspx

• Papper, R., Holmes, M., Popovich, M., Bloxham, M. (2005). Middletown Media Studies 2 Report

Package: Media Day and Concurrent Media Exposure. Center for Media Design, Ball State University.

Available for $750 at http://www.bsu.edu/webapps2/cmdreports/product_select.asp?product_id=10

• Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. A Diary Study of Mobile Information Needs.

Proceedings of CHI ’08 (Florence, Italy, April 2008), ACM Press, 433-442.

• Stinson, J. N., Petroz, G. C., Tait, G., Feldman, B. M., Streiner, D., McGrath, P.J., & Stevens, B.J. e-

Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents with Arthritis. The Clinical

Journal of Pain, 2006, Volume 22 (3).

http://cms.bsu.edu/Academics/CentersandInstitutes/CMD/InsightandResearch/Capabilities/ProjectGallery/MiddletownMediaStudies/MiddletownMediaStudiesI.aspx

http://cms.bsu.edu/Academics/CentersandInstitutes/CMD/InsightandResearch/Capabilities/ProjectGallery/MiddletownMediaStudies/MiddletownMediaStudiesI.aspx

http://www.bsu.edu/webapps2/cmdreports/product_select.asp?product_id=10

http://www.bsu.edu/webapps2/cmdreports/product_select.asp?product_id=10

36

Thank you! Questions?

Comments?

Suggestions?

Technology

Diary Studies in HCI & Psychology