Measuring survey usability

and respondent fatigue

Survey design and usability

Erik KotoCEO

QuestionPro

• We used TryMyUI to test QuestionPro UI

• Discovered a major usability problem

• “Cool!”

• “I wonder if our customers could use this?”

Usability testing: A true story

Survey usability

• Fatigue: length, repetitiveness, or overwhelming complexity

• Clarity: poor or misleading word choice and question phrasing

• Answerability: unsatisfactory or incomplete answer options

• Poorly wording, misspellings, incorrect grammar, dumb mistakes

• Questions which don't quite mean what the researcher intended

• Questions which don't probe to find out what the respondent really meant

• ‘Double barrel’ or multipurpose questions

• Questions without an answer option suitable to the respondent

• Questions which repeat what has already been asked

• Questions which have been missed completely

Common problems

Error blindness

What is the error?

• Best practices help, but won’t

catch errors

• How does design affect data quality?

• Length

• Number of open ended questions

• Rows in matrix table

• Page breaks

• Progress bar

• Intro & instructions

• Number of required fields

• Number of answer options per question

Can we quantify usability?

Introducing

The Survey Respondent Score

Jeff SauroFounding Principal

MeasuringU

1. Create a list of items

2. Test

3. Winnow

4. Assess Reliability & Validity

Steps to Psychometric Validation

Overall UsabilityThis survey was easy to take.In terms of flow, this survey flowed well.Compared to other surveys I have taken, this survey was average.

FatigueThis survey was quick-paced.This survey took average time to complete.If I were asked, I would be likely to take this survey again.Throughout the survey, I felt engaged.

AnswerabilityAll of the questions could be answered accurately.With the answers provided, I was able to answer all of the questions.Did you have enough information to answer the questions?

ClarityThe questions were clear.I could understand the questions easily.To understand the questions, I had to work hard.

13 Candidate Items

Testing The Items

511 Participants 5 Surveys

Winnow: ITEM total Correlations

3 items with low item-total correlations were removed, leaving 10 items with strong internal reliability (Cronbach Alpha = .93)

Initial Factor analysis

One Factor Solution

A scree plot of the eigenvalues suggests a one factor solution. A parallel analysis also shows one factor with eigenvalues greater than those from randomly simulated matrices.

Refining the Items Using IRT

An examination of the items using a logit-transformation reveals some items have a similar discrimination profile, suggesting redundancy.

• Did you have enough information to answer the questions?

• I could understand the questions easily.

• The questions were clear.

Redundant Items

Room for more difficult Items

2 Factor Structure (OBLIQUE)

15

A 2 Factor structure provides at least two items per factor. All items loaded high (>.5)—although factor 2 provides only a negligible amount of variance.

Factor

Fatigue Clarity

In terms of flow, this survey flowed well. 0.84 -0.64

Compared to other surveys I have taken, this survey was average. 0.84 -0.60

Throughout the survey, I felt engaged. 0.81 -0.62

This survey was easy to take. 0.77 -0.68

If I were asked, I would be likely to take this survey again. 0.77 -0.65

I could understand the questions easily. 0.68 -0.92

The questions were clear. 0.72 -0.88

% of Variance Explained 68% 9%

Extraction Method: Principal Axis Factoring.

Rotation Method: Oblimin with Kaiser Normalization.

Reliability & Validity

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2 3 4 5

SR

S

Survey Number (Hardest to Easiest)

High-reliability (Cronbach alpha = .93) and able to discriminate well across surveys

Validity

The SRS correlated highly with survey abandonment rate & total number of questions (r >.8)

Abandonment Rate

Number of Questions

Predictive Validity & Future Research

18

7 Item SRS

This survey was easy to take.In terms of flow, this survey flowed well.Compared to other surveys I have taken, this survey was average.If I were asked, I would be likely to take this survey again.Throughout the survey, I felt engaged.The questions were clear.I could understand the questions easily.

Predicting Response Rates & Continuing to Refine

3 Factor Structure (Orthogonal)

A 3 Factor structure provides at least two items per factor. All items loaded high (>.5).

Rotated Factor Matrixa

Fatigue Clarity Answerability

Compared to other surveys I have taken, this survey was average. 0.77 0.25 0.23

In terms of flow, this survey flowed well. 0.71 0.27 0.27

Throughout the survey, I felt engaged. 0.67 0.26 0.31

If I were asked, I would be likely to take this survey again. 0.61 0.34 0.33

This survey was easy to take. 0.59 0.38 0.33

I could understand the questions easily. 0.34 0.77 0.34

The questions were clear. 0.39 0.71 0.37

With the answers provided, I was able to answer all of the questions. 0.36 0.36 0.76

All of the questions could be answered accurately. 0.39 0.36 0.69

% of Variance Explained 65% 9% 6%

Forcing a 4 Factor Structure (Oblique)

Given that factors were likely to be correlated, an exploratory factor analysis using principal axis factoring with oblique rotation (direct oblimin) was then conducted and four factors were retained. All items loaded high (>.5) on at least one factor but one item cross-loaded high on two factors and could be removed.

Structure Matrix

Factor

Fatigue Clarity

Answer-

ability Usability

Compared to other surveys I have taken, this survey was average. 0.84 -0.60 -0.59 -0.59

Throughout the survey, I felt engaged. 0.82 -0.60 -0.49 -0.62

In terms of flow, this survey flowed well. 0.79 -0.60 -0.61 -0.60

If I were asked, I would be likely to take this survey again. 0.74 -0.64 -0.63 -0.65

The questions were clear. 0.68 -0.94 -0.55 -0.74

I could understand the questions easily. 0.64 -0.86 -0.55 -0.72

This survey was easy to take. 0.71 -0.65 -0.94 -0.64

With the answers provided, I was able to answer all of the questions. 0.67 -0.73 -0.54 -0.91

All of the questions could be answered accurately. 0.67 -0.72 -0.53 -0.87

Did you have enough information to answer the questions? 0.66 -0.80 -0.51 -0.81

Extraction Method: Principal Axis Factoring.

Rotation Method: Oblimin with Kaiser Normalization.

High Cross

Loading

Forced 4 Factor Structure (Orthogonal)

A varimax rotated factor analysis was also conducted showing the same factor structure as the oblique rotation. All items loaded high (>.5).

Rotated Factor Matrixa

Factor

Fatigue Clarity

Answer-

ability Usability

Compared to other surveys I have taken, this survey was average. 0.73 0.24 0.22 0.27

Throughout the survey, I felt engaged. 0.71 0.26 0.29 0.13

In terms of flow, this survey flowed well. 0.64 0.26 0.25 0.31

If I were asked, I would be likely to take this survey again. 0.52 0.31 0.32 0.36

I could understand the questions easily. 0.30 0.74 0.33 0.23

The questions were clear. 0.35 0.72 0.35 0.23

With the answers provided, I was able to answer all of the questions. 0.32 0.35 0.74 0.23

All of the questions could be answered accurately. 0.35 0.35 0.68 0.22

This survey was easy to take. 0.39 0.29 0.27 0.77

Extraction Method: Principal Axis Factoring.

Rotation Method: Varimax with Kaiser Normalization.

a. Rotation converged in 7 iterations.

QuestionPro Integration with TryMyUI

Ritvij GautamCEO

TryMyUI

• Crowdsourced, User-Interface usability testing tool.

• Traditionally when people hear UI/usability testing, they think of Websites & Mobile

Apps/Web-apps.

• ANYTHING which involves a user has a UI that needs testing.

• How do we leverage our tool to test something like a survey?

TryMyUI’s Challenge

• QuestionPro is a comprehensive survey

tool that allows its customers to ask a

variety of questions.

• Now we need to make sure you ask the

right ones.

Wait, is this REALLY a problem?

How do you get actionable feedback on a feedback mechanism? Or meta-feedback?

1. Break it down to the elemental vectors of Cognitive Stress induced by surveys.

2. Have a psychometrically validated meta-survey to assess cognitive stress along these vectors.

3. Cross-reference the data with video of a demographically screened respondent taking the survey.

4. GET THE USER’S VIEW!

The Solution

Ordering the Test

Results Received

Watch the Videos

Review the SRS

Thank you!

Erik KotoCEO

www.questionpro.com

Jeff SauroFounding Principal

MeasuringU.com

Ritvij GautamCEO

TryMyUI.com