You Want me to What? A Practical Guide to Diary Studies

What is a Diary Study?

Definition*: “Research method that involves providing participants with the materials and structure to record daily events, tasks and perceptions around a given subject in order to gain insight into their behaviour and needs over time.”

*http://www.usability.gov/what-and-why/glossary/diary-study.html

Mike’s Study

Purpose: to examine an existing system to prepare for a global redesign

Why a diary study?

•  To address the difficulty of determining the impact of known usability issues during onboarding

•  What prevents a motivated user from forming patterns of use?

•  Examined patterns of use, sticking points, goals and motivations

Vicky’s Study

Purpose: Understand how different user types work with an Email Marketing System over time.

Why a diary study:

●  Diary study enabled us to track user actions and satisfaction level OVER TIME – what changes for the user as they become more skilled?

●  Factors we were gauging: blockers to success, satisfaction points, business goals and motivations, which features were used most frequently

Liz’s Study Purpose: Gain an understanding of the experience of current customers when they are

upgraded to a newer version of the product. Why a diary study: ●  Study conditions recreate the context for transitioning users as closely as possible ●  Can gauge how quickly participants adjust to the new version ●  Participants have the freedom to report any type of issue or concern ●  Longer timeline distinguishes initial learning from persistent issues

What can you learn from a Diary Study?

It can provide a better understanding of usage patterns over time: •  Learnability •  Acceptance of a product or product features •  Issues or challenges that impede usage

Generally: most any type of change in behavior or attitude

Advantages of Diary Studies

•  Authenticity of information collected: –  User’s environment rather than a lab setting –  Participants feel safer so they’re more likely to be open –  Can get past initial learning

•  Benefits for participants: –  Less intrusive –  More flexible (e.g., scheduling)

•  Can uncover unexpected issues

Disadvantages & Challenges

•  Overall: –  Time- and resource-intensive –  Logistically unwieldy –  Less control over study environment

•  Participants: –  Recruitment, Management, & Retention

•  Data: –  Self-reported, but use the resources available to you –  Overwhelming!

A Practitioner’s Checklist:

❏  Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results

Designing the Study

Managing Length of Study •  Long enough to capture desired

behaviors •  Stay meaningful for participants and

stakeholders •  Reflect users’ natural work cadence as

much as possible

Designing the Study

Components •  Ongoing feedback on user experience

–  Diary entries •  Interim check-ins

–  Surveys / interviews •  Final review touch point

–  Follow-up interviews

Designing the Study

Unstructured •  Allows for more unexpected, possibly more realistic data

•  BUT, may not get desired data, and lots of manual labor sorting through results

Structured •  Less time consuming data analysis, higher probability of

getting relevant data

•  BUT, may not reflect user’s natural workflow, and a lot of dictating may feel too formal and restrictive to users

Example for study design - Diary Questions

•  Name

•  What did you do with the tool today?

•  What did you want to accomplish?

•  Did you accomplish what you wanted?

•  What specific features did you use?

•  What did you like about your experience today?

•  What did you not like about your experience today?

•  What else would you like us to know?

•  Screen shots (optional)

Example for study design - Setup

“Is the tool meeting your business goals?” “What features have you used?” “Which features do you like most / least?”

“How has your experience changed since 3 weeks ago?” “Have you tried any new features?” “Do you feel more skilled than you did 3 weeks ago?”

Example for study design - Phone Interviews

•  Follow-up on diary entries and surveys

•  Kept conversation less structured •  Gave deeper understanding of user

goals, issues, etc. •  Benefit of hearing user’s tone

(excitement or frustration)

A Practitioner’s Checklist:

✔ Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results

Recruiting - “Who’s Coming with me?!”

Recruiting

Be realistic about how long this is likely to take •  It will probably take a lot longer than you think to hit your number •  Be prepared to be flexible, prioritize screener criteria if necessary

Strategies for minimizing attrition:

•  Graduated payments –  Have payments tied to something you can control (e.g., interviews)

•  If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it

A Practitioner’s Checklist:

✔ Designing the Study ✔ Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results

Participant management

Managing Participants

•  Make it as easy as possible for them –  Online data entry –  Include links in every communication –  Have some structure (not completely open-ended)

•  Give participants permission to be brief and/or repeat themselves, and let them be themselves!

–  Remember, it’s about a truer contextual perspective

•  Guard against being too invasive –  Let participants choose the frequency of reminders –  Make it explicit that reminders are automated

•  Monitor for compliance and to answer questions or redirect to support

A Practitioner’s Checklist:

✔ Designing the Study ✔ Recruiting ✔ Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results

Managing the Data

There are tons of great online resources -- make use of them! •  Online form for diary entries

–  Easy for participants to access –  Researcher has access to data in real time to monitor

•  Online self-scheduler for interviews/checkins •  Online form for data entry (e.g., for interviews) •  Survey tools for questionnaires, rating questions etc.

But watch out for:

•  Privacy issues •  Appropriateness for your participants

A Practitioner’s Checklist:

✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ❏  Analyzing the Data ❏  Reporting the Results

Analyzing the Data

Analyzing the Data

It’s largely a manual process: 1.  Read through the data looking for issues, patterns etc. 2.  Make a list of tags and cross reference with your data 3.  Repeat

Qualitative data analysis tools: •  Lots of them available •  Traditionally have been more helpful for ongoing data collection needs than

for a one-time study •  Worth keeping an eye out

Analyzing the Data

Strategies: •  Try to Identify categories and tags in advance as part of defining

the study •  “Find” is your friend but watch out for:

–  misspellings –  different word forms –  different words for the same/similar concepts –  different meanings for the same words

•  Have a method for collecting or flagging verbatims as you go •  Use learnings from other touch-points to help identify tags &

categories

Analyzing the Data

Don’t wait until the end! •  Keep the job manageable •  Keep analysis timely •  Allow for mid-course corrections

Include some selected-response measures: •  Easy to analyze •  Consistent across participants and over time •  Provide metrics to supplement open-ended responses Look for the surprises!

My Approach Two spreadsheets:

1.  Raw data: diary entries, numbered chronologically 2.  “Processed” data, with columns for:

1.  Tags 2.  Entry numbers 3.  Verbatims

Same 3 Likert questions in all diary entries: •  “easy” results

My approach: Get your participants to help you!

Two weeks before the final interview: •  Created individual spreadsheets •  Asked participants to review and think about:

1.  Their overall experience making the transition 2.  What they liked best 3.  What was the biggest challenge 4.  What we could have done to make the transition easier

DIY vs. DIFM

•  Emerging tools like dscout can reduce overhead –  The trade-off can be

flexibility –  Decide what you want to

accomplish with your methodology before you approach any tools

•  We were curious, so we’re running a study right now with thanks to our UXPA organizers!

A Practitioner’s Checklist:

✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ❏  Reporting the Results

Reporting the Results Changes over time •  Learnability

–  (Qual) Frequency with which issues or problems are reported –  (Quant) How easy/difficult was it for you to meet your goals in this session? To

what extent were you able to meet your goals in this session?

•  Long-term patterns of use (what features remain important over time)

–  (Qual) Changes in users’ stated goals –  (Quant) % of use of different features

•  Changes in user opinions over time –  (Qual) % of comments classified is positive/negative/neutral –  (Quant) How satisfied/dissatisfied are you with the product?

Reporting the Results

First Portion Second Portion Third Portion

Reporting the Results

A Practitioner’s Checklist:

✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ✔ Reporting the Results

Lessons Learned

Mike: 1.  Planned vs. Unplanned learnings (be prepared to dive back in) 2.  It all comes down to Codifying open responses 3.  Visualize learnings for broad adoption

Vicky:

1.  Try to start all participants at the same time 2.  Automate as much as possible 3.  Review the data as you go - don’t wait until the end

Liz: 1.  Structure can be your friend 2.  Control what you can but don’t try to control what you can’t 3.  Embrace the chaos!

Questions?

References

1.  http://www.uxbooth.com/articles/jumpstart-design-research-with-a-diary-study/

2.  http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/

3.  https://uxmag.com/articles/making-the-most-of-ethnographic-research

4.  http://www.spotless.co.uk/insights/6-things-for-ux-diary-study

5.  http://www.uxbooth.com/articles/dear-diary-hard-say-goodbye/

6.  https://blinkux.com/blog/dear-digital-diary-a-powerful-tool-in-user-research/

7.  http://uxpamagazine.org/dear-diary/

8.  http://www.slideshare.net/jaremfan/the-goodness-of-diary-studies