[IxD] Week 08. Data Gathering

Lecture 7

Data Gathering

Interaction Design / IID 2017 Spring Class hours : Wednesday 2:00 pm – 5:50 pm Lecture room : International Campus Veritas Hall B308 26th April

DATA GATHERING

Chapter 7

Lecture #7 IID_Interaction Design 2

Objectives

• Discuss how to plan and run a successful data gathering program.

• Enable you to plan and run an interview.

• Enable you to design a simple questionnaire.

• Enable you to plan and carry out an observation.


Five key issues

• Setting goals

– Decide how to analyze data once collected

• Identifying participants

– Decide who to gather data from

• Relationship with participants

– Clear and professional

– Informed consent when appropriate

• Triangulation

– Look at data from more than one perspective

– Collect more than one type of data, eg qualitative from experiments and qualitative from

interviews

• Pilot studies

– Small trial of main study


Data recording

• Notes, audio, video, photographs can be used individually or in

combination:

– Notes plus photographs

– Audio plus photographs

– Video

• Different challenges and advantages with each combination


Interviews

• Unstructured - are not directed by a script. Rich but not replicable.

• Structured - are tightly scripted, often like a questionnaire. Replicable but

may lack richness.

• Semi-structured - guided by a script but interesting issues can be

explored in more depth. Can provide a good balance between richness

and replicability.

• Focus groups – a group interview


Interview questions

• Two types:

– ‘closed questions’ have a predetermined answer format, e.g.. ‘yes’ or ‘no’

– ‘open questions’ do not have a predetermined format

• Closed questions are easier to analyze

• Avoid:

– Long questions

– Compound sentences - split them into two

– Jargon and language that the interviewee may not understand

– Leading questions that make assumptions e.g.. why do you like …?

– Unconscious biases e.g.. gender stereotypes


Running the interview

• Introduction – introduce yourself, explain the goals of the interview, reassure about

the ethical issues, ask to record, present the informed consent form.

• Warm-up – make first questions easy and non-threatening.

• Main body – present questions in a logical order

• A cool-off period – include a few easy questions to defuse tension at the end

• Closure – thank interviewee, signal the end,

eg. switch recorder off.


Enriching the interview process

• Props - devices for prompting interviewee, e.g. use a prototype,

scenario


Questionnaires

• Questions can be closed or open

• Closed questions are easier to analyze, and may be distributed and

analyzed by computer

• Can be administered to large populations

• Disseminated by paper, email and the web

• Sampling can be a problem when the size of a population is unknown

as is common online evaluation


Questionnaire design

• The impact of a question can be influenced by question order.

• You may need different versions of the questionnaire for different populations.

• Provide clear instructions on how to complete the questionnaire.

• Strike a balance between using white space and keeping the questionnaire compact.

• Avoid very long questionnaires

• Decide on whether phrases will all be positive, all negative or mixed.


Question and response format

• ‘Yes’ and ‘No’ checkboxes

• Checkboxes that offer many options

• Rating scales

– Likert scales

– semantic scales

– 3, 5, 7 or more points

• Open-ended responses


Encouraging a good response

• Make sure purpose of study is clear

• Promise anonymity

• Ensure questionnaire is well designed

• Offer a short version for those who do not have time to complete a

long questionnaire

• If mailed, include a stamped addressed envelope

• Follow-up with emails, phone calls, letters

• Provide an incentive

• 40% response rate is good, 20% is often acceptable


Advantages of online questionnaires

• Relatively easy and quick to distribute

• Responses are usually received quickly

• No copying and postage costs

• Data can be collected in database for analysis

• Time required for data analysis is reduced

• Errors can be corrected easily


Example of an online questionnaire


Figure 7.8 An excerpt from a web-based questionnaire showing check boxes, radio buttons, and pull-down menus

Problems with online questionnaires

• Sampling is problematic if population size is unknown

• Preventing individuals from responding more than once can be a

problem

• Individuals have also been known to change questions in email

questionnaires


Observation

• Direct observation in the field

– Structuring frameworks

– Degree of participation (insider or outsider)

– Ethnography

• Direct observation in controlled environments

• Indirect observation: tracking users’ activities

– Diaries

– Interaction logging

– Video and photographs collected remotely by drones or other equipment


Observation


Figure 7.9 Mars Exploration Rover

Structuring frameworks to guide Observation

• Three easy-to-remember parts:

– The person: Who?

– The place: Where?

– The thing: What?


Structuring frameworks to guide Observation

• A more detailed framework (Robson, 2014):

– Space: What is the physical space like and how is it laid out?

– Actors: What are the names and relevant details of the people involved?

– Activities: What are the actors doing and why?

– Objects: What physical objects are present, such as furniture

– Acts: What are specific individual actions?

– Events: Is what you observe part of a special event?

– Time: What is the sequence of events?

– Goals: What are the actors trying to accomplish?

– Feelings: What is the mood of the group and of individuals?


Planning and conducting observation in the field

• Decide on how involved you will be: passive observer to active

participant

• How to gain acceptance

• How to handle sensitive topics, eg. culture, private spaces, etc.

• How to collect the data:

– What data to collect

– What equipment to use

– When to stop observing


Ethnography (1)

• Ethnography is a philosophy with a set of techniques that include

participant observation and interviews

• Debate about differences between participant observation and

ethnography

• Ethnographers immerse themselves in the culture that they study

• A researcher’s degree of participation can vary along a scale from

‘outside’ to ‘inside’

• Analyzing video and data logs can be time-consuming

• Collections of comments, incidents, and artifacts are made


Ethnography (2)

• Co-operation of people being observed is required

• Informants are useful

• Data analysis is continuous

• Interpretivist technique

• Questions get refined as understanding grows

• Reports usually contain examples


Ethnography (3)


Figure 7.10 (a) The situation before MERboard; (b) A scientist using MERboard to present information Source: J. Trimble, R. Wales and R. Gossweiler (2002): “NASA position paper for the CSCW 2002 workshop on Public, Community and Situated Displays: Merboard”.

Online Ethnography

• Virtual, Online, Netnography

• Online and offline activity

• Interaction online differs from face-to-face

• Virtual worlds have a persistence that physical worlds do not have

• Ethical considerations and presentation of results are different


Observations and materials that might be collected (Crabtree, 2007)

• Activity or job descriptions.

• Rules and procedures that govern particular activities.

• Descriptions of activities observed.

• Recordings of the talk taking place between parties.

• Informal interviews with participants explaining the detail of observed activities.

• Diagrams of the physical layout, including the position of artifacts.

• Other information collected when observing activities:

– Photographs of artifacts (documents, diagrams, forms, computers, etc.)

– Videos of artifacts.

– Descriptions of artifacts.

– Workflow diagrams showing the sequential order of tasks.

– Process maps showing connections between activities.


Observation in a controlled environment

• Direct observation

– Think aloud techniques

• Indirect observation – tracking users’ activities

– Diaries

– Interaction logs

– Web analytics

• Video, audio, photos, notes are used to capture data in both types of

observations


Web analytics

• A system of tools and techniques for optimizing web usage by:

– Measuring,

– Collecting,

– Analyzing, and

– Reporting web data

• Typically focus on the number of web visitors and page views.


A section of Google analytics dashboard for id-book.com


Figure 7.14 Segments of the Google Analytics dashboard for id-book.com in September 2014 (a) audience overview, (b) screen resolution of mobile devices used to view the website

A section of Google analytics dashboard for id-book.com


Figure 7.14 Segments of the Google Analytics dashboard for id-book.com in September 2014 (a) audience overview, (b) screen resolution of mobile devices used to view the website

Choosing and combining techniques

• Depends on the:

– Focus of the study

– Participants involved

– Nature of the technique(s)

– Resources available

– Time available


Summary

• Data gathering sessions should have clear goals.

• An informed consent may be needed.

• Five key issues of data gathering are: goals, choosing participants, triangulation,

participant relationship, pilot.

• Data may be recorded using handwritten notes, audio or video recording, a camera,

or any combination of these.

• Interviews may be structured, semi-structured or unstructured

• Focus groups are group interviews

• Questionnaires may be on paper, online or telephone

• Observation may be direct or indirect, in the field or in controlled settings.

• Techniques can be combined depending on the study focus, participants, nature of

technique, available resources and time.


DESIGNING FOR THE DIGITAL AGE: HOW TO CREATE HUMAN-CENTERED PRODUCTS AND SERVICES CHAPTER 9. OTHER SOURCES OF INFORMATION AND INSPIRATION

Workshop #1


When You Have Less Time

• When possible, try to squeeze in at least one or two days of user

interviews.

– It's true that you risk getting unusual interview participants who could

skew your thinking, but this is rare, and the risk is limited as long as you

compare what you see to what you're hearing from stakeholders.

– Focus on the kinds of users who are most critical to the product's success

or the ones you suspect are the least understood.

– If you're not allowed to recruit users on your own, use the friends-and-

family method


When You Have Less Time

• A larger interview sample allows for some quantitative analysis of the

data, but is unlikely to result in better design.

• Any existing focus group or survey data may also be useful.

– Don't limit your requests for such information to the product team;

corporate marketing groups frequently conduct research that isn't

disseminated.

– Product managers or professional services staff may also have trip notes

from customer site visits that are worth perusing.

– If possible, review these before the stakeholder meeting so you can use

the data to move things forward.


When You Have More Time

• A larger interview sample allows for some quantitative analysis of the data, which

can be persuasive or comforting to many people.

– However, a larger sample adds considerable analysis time and is unlikely to result in

better design, except to the extent that it lets you catch an edge case you otherwise

might have missed.

• A long session should include an initial interview using the techniques described

in Chapter 7, followed by some mixture of quiet observation time with opportunities

to ask more questions about what you observe.

– If activities cannot be interrupted, use a small video camera; replay interesting

segments for the interviewee later so you can ask questions about specifics. Consider

whether it's better to have the whole team observing or to have just one person follow

the informant with a video camera; surgeons are accustomed to working with large

audiences, but most office workers are not.


Supplemental Research Methods

• Public-space observation


Figure 9.1. If you were designing wayfinding or an information kiosk for an airport, it would be informative to hang out at the information desk for a while.

Exercise

• Would you be comfortable conducting research in the following ways?

Why or why not?

1. You're designing a café it acceptable to buy coffee in a competitive café,

then sit at a table and watch what people do there, how long they stay,

and what seems to encourage them to spend money?

2. You're designing a children's game. Is it acceptable to sit near an

elementary school and watch how children play? How about wandering

over and asking questions of some of the children? What about looking at

their pages on MySpace or another social networking site?


Exercise

3. You're attempting to improve the patient experience at a hospital. Is it acceptable to sit

in a waiting room observing how people behave there?

4. You're designing a patient Web site for a new diabetes drug. Is it acceptable to sit in an

endocrinologist's waiting room and chat with people about their conditions, not

claiming to have diabetes but also not identifying yourself as a researcher?

• Provided you're comfortable doing so, conduct some informal observations in a

public space for the LocalGuide or RoomFinder. What behaviors do you see in this

context that you did not see (or don't think you would have seen) in an interview?

What did you see that provided an interesting insight?



• Mystery shopper

– A variation on public-space observation is being a "mystery shopper," someone

who tries out the customer experience without being identified as a

researcher.

– Anonymous usage is an important research tool because the story you get

during explicit research may be sanitized; this is why good restaurant

reviewers don't tell the staff who they are.

– This technique is a powerful tool for convincing executives of the need for

change; many become passionate advocates for improvement after seeing their own products or services through a customer's eyes. I'm convinced

that more products and services would be better if more executives, as one of

my clients put it, "ate their own dog food."


Exercise

• If you work for a company that sells products or provides customer

support, try using the service as an ordinary customer.

1. Alternatively, try using a service you're not familiar with but might

ordinarily use anyway (such as a physical store, online service, or

telephone customer support center).

2. What makes you feel good about the service and the company providing

it?

3. What gives you a negative opinion?



• Diaries

1. In interviews, it can be difficult to get a sense of behavior over time

because you have to rely on the participant's memory of past activities or

circumstances, and artifacts can only do so much to prompt that.

2. One way to widen your view of someone's activities without shadowing

them 24/7 is to ask them to keep a diary.

3. This can be somewhat structured, much like a survey taken several times,

or can be free-form entry guided by a few questions.



4. A diary can take almost any form: written responses to a periodic e-mail

reminder, a handwritten notebook, a narrated video, or photos with

written commentary.

5. Keep in mind that a diary has limitations: Self-reporting error is likely. If

possible, sit down with each participant and conduct a follow-up

interview using the diary as a basis for questions.



• Surveys

– Numbers helpful in disabusing stakeholders of pet beliefs.

– My opinion alone wouldn't have been persuasive, but quantitative data

helped stakeholders see that they'd be limiting the experience of their

most important audience.


Figure 9.2. An example of a survey with a Likert scale.


• Surveys (continues)

– FINDING EXISTING SURVEY DATA

– DEVELOPING YOUR OWN SURVEY

• Step 1: Identify your audience and goals

• Step 2: Craft questions and instructions

• Step 3: Determine your sample size

• Step 4: Decide how to recruit participants



• Step 1: Identify your audience and goals

– First, work with your teammates (and probably the stakeholders) to

determine what you're trying to learn.

– Is this a single survey to understand demographics and attitudes to inform

interview planning?

– Are you trying to assess how many people fit the behavior patterns you've

already observed?

– Or are you trying to assess the impact of your design with a before and

after comparison?



• Step 2: Craft questions and instructions

– Ask only one question at a time. For example, "If you own a camera, who is

the manufacturer?" is really asking two questions. It would be better to ask

whether the participant owns a camera, then ask who the manufacturer is

only if the first answer is yes.

– Be specific. "Do you ever use the Web?" won't really tell you much, and a

respondent who's used it once may not know how to answer. "How often do

you use the Web? Daily / Two or more times a week / Two or more times a

month / Less than twice a month / Never" would be better. Offer quantity or

frequency choices with specific numbers, since terms like "often" and "seldom"

are relative.



– Make the options for any single-answer question mutually exclusive. For example, if

you're asking about income, don't have a $50,000-$60,000 category and a $60,000-$70,000

category, because the person whose salary is $60,000 won't know what to pick; $50,000-

$59,999 is better.

– Make lists as complete as possible. If someone who drives a minivan has to choose

either "car" or "truck," she will either abandon the survey in frustration or be forced to

choose an answer that doesn't really fit, thereby introducing error.

– Allow for participants who can't provide definitive answers. Include options such as

"other," "not applicable," or "I don't know" when possible, so you don't get incorrect

answers or cause people to quit in frustration.



– Avoid negative construction for multiple choice questions. For

example, "which of the following do you not use" is likely to be read as

"Which of the following do you use." If you absolutely can't avoid it,

visually or verbally emphasize the negative word.

– Limit the options in a Likert scale to five. A scale of 1 to 5 lets people

differentiate between good and great, but a scale of 1 to 10 causes

confusion. Having an odd number of choices allows for a neutral answer,

which generally provides a more accurate picture of attitudes. Make sure

the high/low or positive/negative values are always at the same end of the

scale.



• Use both positive and negative phrasing within your sample. When asking if

people strongly agree or strongly disagree whether something is good, help

minimize bias due to a desire to please by using positive phrasing with half the

group and negative phrasing with the other half. For example, if you present a

Likert scale with a statement like, "Product X is affordable," you will get an

artificially high level of agreement. Balance this by phrasing it as "Product X is

expensive" for the other half of the group.

• Vary list sequences. If possible, vary the sequence of items in lists; this helps

minimize any bias due to people picking things at the top of the list and skipping the

rest. Don't randomize the elements in any list that has a natural progression to it,

such as income or age ranges.



• Step 3: Determine your sample size

– https://www.surveymonkey.com/mp/sample-size-calculator/


Size of population 5% Error 90% Confidence

5% Error 95% Confidence



1,000 214 278 399 648

10,000 265 370 622 1556

100,000 272 383 659 1810

1,000,000 272 384 663 1840

10,000,000 272 384 664 1843

Table 9.1. Sample size examples.

https://www.surveymonkey.com/mp/sample-size-calculator/


• Step 4: Decide how to recruit participants

– Once you've identified how many people you need to recruit, you have to

decide where to find them and how to invite them to participate.

– Many companies recruit users online through invitations on discussion lists,

pop-up invitations on their own Web sites, or e-mail invitations to existing

customers.

– Inviting only your existing customers limits you in two ways. First, it won't tell

you anything about the people you're not reaching. Second, it tends to bias the

results toward favorable responses, since the least happy people have

probably switched to a competitor.

– Inviting only ex-customers provides an unfavorable bias, since they're likely to

be unhappy.



– A mix of current customers, former customers, and those who have never

been customers is the best bet for most studies; proportion depends on the

goal of the survey.

– Inviting people using e-mail, discussion groups, and Web sites is fine if you're

targeting people who spend much time online, but could be missing an

important segment.

– Telephone, advertising, or direct mail may work best in some circumstances.

However, opt-in invitations, such as permanent links on a Web site, may

create a bias toward extreme views, since people who are very happy or very

unhappy are the most likely to seek out those opportunities.

– Opt-out invitations, such as pop-ups and phone calls, minimize this bias.



• Step 5: Decide when and for how long to conduct the survey

– The timing and duration of your survey can have a tremendous impact on your

sample.

– Certain audiences are either more or less available at particular times of year,

days of the week, or times of day.

– If you conduct a telephone survey or in-store study during a weekday, your

results will be skewed toward people who don't work outside the home.

– Even online, some people may be less likely to spend time on a survey during

the work week or during a busy season. Make sure your data collection

window spans enough time to gather data from people with varied habits.


Exercise

• Imagine that the stakeholders for the LocalGuide (see Chapter 6) are

trying to determine how many of a possible ten million business

travelers are likely to use it.

– What questions would help you understand this?

– Develop a survey.

– How large does the sample need to be, assuming a high degree of

confidence is critical?

– How would you recruit?



• Web analytics and customer support data

– When during the day people are visiting your site or particular pages

– How often people visit (if you use identity cookies)

– How long people stay on the site or particular pages

– What percentage of users makes purchases (if applicable)

– Which types of customers are worth the most revenue

– What percentage doesn't purchase the items in their shopping carts

– Where frequent page errors occur

– What sites are referring people to yours



– What terms people entered into a search engine if it's the referring page

– What people search for on your site

– What operating systems and browsers people are using

– The most typical paths through the site (though cached pages can throw

this off)


Figure 9.3. Web analytics can help you identify where there might be a problem, but won't explain why the problem occurs.


• Focus groups

– A focus group is a facilitated, usually 60- to 90-minute meeting with anywhere from five

to a dozen members of a target market.

– The best use for focus groups is when you have a new product idea and know very little

about the people you think might buy and use it.

– If you do plan to conduct a focus group, begin by defining what you want to learn.

• Industry trends?

• General work processes and relationships among roles?

– Consider whether it's more interesting to get the range of views within a particular set of

people or to see how views differ across roles or perspectives. It's generally a good idea

to conduct two to four similar groups; as with interviews, this helps you see if one group

could be an outlier.



• Card sorting

– Card sorting is best suited to relatively small Web sites with users who

understand most of the content. Sorting will help you see that some users

organize mentally based on one criterion while others start from a

different entry point altogether.


Figure 9.4. In card sorting, participants arrange and sometimes edit category names to fit their mental models.


• Competitive products and services

– Companies that spend more time analyzing their competitors than

understanding their customers are likely to be followers rather than

market leaders. That said, a good designer should spend enough time on

the competitors to understand their vulnerabilities and any opportunities

for differentiation.

• Literature and media

– Fiction is also a way for designers to access the zeitgeist that affects how

people respond to new technology products and new interaction

paradigms.


Research Design

• Participants Sampling

– Random Sampling, Stratified Sampling, Volunteer Sampling, Opportunity

Sampling

• Research Questions

– IV >> DV

• Measurements

– Performance, Satisfaction.

• Research Methods

– Interviews, Survey, Diary, Observation, Ethnography

• Stimuli Development

– Iterative Development


Homework


[Team] Research Design

[Team] Data Gathering

[Team] Data Analysis, Interpretation, and Presentation

1 2 3

Steps - Participants

Sampling - Research Questions - Measurements - Research Methods - Stimuli

Development

Steps - Set a goal - Recruit participants - Conduct data gathering by

- Interviews - Questionnaires - Observations

Outcomes - Raw data

- Transcripts - Voice recordings - Video Recordings

- Coding - Open coding - Axial coding - Selective Coding

- Models and possible quantitative Analysis

- Presentation

Submission Due : 11: 59 pm Sun. 30th April

Education

[IxD] Week 08. Data Gathering