Using AI to Make Sense of Customer Feedback

Using AI to Make Sense of

Customer Feedback

Alyona Medelyan

@zelandiya

Correct Understanding of Customer Feedback

Can Save Millions

2015: Tens of Thousands of New Zealanders

were Surveyed About the new Flag

Government Reported

the Results of Manual Feedback Analysis

Actual Responses

Two costly & unnecessary referendum followed. Outcome: NZ kept the current flag

Millions could have been saved!

People wanted to ”keep the current flag”

1. Types of customer feedback

2. Why analyzing customer feedback is important

3. Why is it hard

4. Approaches

5. Applying AI to customer feedback analysis

6. Demo

Different Types

of Customer Feedback

Types of Customer Feedback

one-on-one interviews / focus groups

call centre logs / complaints

social media

open-ended survey questions / reviews

quantitate survey questions

UX tests / analytics

unstructured

structured

Collection Analysis Insight

one-on-one interviews / focus groups hard hard good

call centre logs / complaints easy hard limited

social media easy hard limited

open-ended survey questions / reviews easy medium good

quantitate survey questions easy easy limited

UX tests / analytics medium easy limited

unstructured

structured

Comparing Types of Customer Feedback

Why Understanding

Customer Feedback

is More Important than Ever

Customer Experience

is the New Marketing

It’s Measured Using

Net Promoter Score Surveys

Image credit

The number of “Net Promoter Score”

searches on Google since 2004

1. Growing Number of

Satisfaction Surveys and Reviews

v

¯\_(ツ)_/¯

2. The Need to Explain

the Why’s Behind the Scores

Net Promoter Score by month over time

3. Scores can be Cheated

Unstructured Feedback, not so Much

Why Analyzing

Customer Feedback is Hard

Common Misconception:

Sarcasm Makes Analysis Hard

One of Many Sarcastic Tui Beer Adverts

Sarcasm is Hard: Even People Struggle

I’ll keep it in

mind

They’ll do itI’ve

forgotten

already

Sarcasm is Rarer Than You Think

Dataset Sarcasm Example

NPS Survey 1%I’m so disappointed! What a great

customer service you have!

Social Media

comments5% Very helpful answer. Troll.

The Actual Challenges

With Customer Feedback

Challenge 1: Messy Data

How many ways there are to say

‘wet paper’?

Challenge 2: Synonyms and Paraphrases

Hundreds of

possible variations

of the same theme

wet

dripping

soaking

soaked

damp

drenched

paper

papers

newspaper

news paper

newspapers

news papers

+

Paraphrasing the Same Theme

Challenge 3: Negation

Positive or Negative?

My coffee was great positive

My coffee was awful negative

My coffee was not great negative

My coffee was not that great neutral?

I did not think my coffee was great negative

I did not expect my coffee to be this great positive

I was disappointed with the quality of the coffee negative

I was not disappointed with the quality of the coffee positive

Approaches to

Customer Feedback Analysis

Manual Coding

1.

Figure out the Code Frame, Apply, Repeat

What is the meaning of life?

1 2 3 4 5

What is the meaning of life?

42

Friends and family

Making a difference in the world

Happiness

Finding happiness

To achieve, to conquer

Family

…

What is the meaning of life?

42

Friends and family

Making a difference in the world

Happiness

Finding happiness

To achieve, to conquer

Family

…

1

2

3

4

4

5

2

Sentiment in a Manual Code Frame

Customer Service

Positive Negative

Timely Nice Helpful Didn’t fix issue Rude

Word Clouds

2.

“Every time I see a word cloud presented as insight,

I die a little inside.”

– J. Harris, journalist

Word Clouds Lack

Interpretation, Context, Meaning

“Overall the language

focuses on sweeping

statements focusing on

the state of the nation.”

Kalev Leetaru (Forbes)

You wouldn’t create a Word Cloud from your Numbers,

why is it ok from Text?

Rule-based Approaches

3.

It’s Hard to Find a Rule That Works Well

I was impressed by how friendly the person

on the other end of the line wasStaff friendliness ✔

The lady who helped me was friendly Staff friendliness ✔

Friendliness of staff Staff friendliness ✔

Your website is very user friendly Staff friendliness ✘

The young man on the phone was very pleasant Other ✘

friendly OR friendliness –> Staff friendliness

Text Categorization

4.

old

customer

responses

categories

new

customer

responses

Machine

Learning

Algorithm

Predictive

Model categories

Need for Sufficient Training Data,

and Clear Categories

Customer Feedback Analysis

Needs to be ‘Unsupervised’

Thanks to an unsupervised approach, Facebook found

Candi Crash Saga causes low App Store reviews

Topic Modeling

5.

21

3

A Topic can be Hard to Interpret

2

???ok

Source: Ben Fields

Sentiment

1. Rule-based (dictionary)

2. Text categorization (positive / negative)

Two Sentiment Detection Approaches

Advances in AI > Customer Feedback

Messy Data

Paraphrases

Negation

AI > Challenges

Word2vec*

Deep Learning

*See also: Conceptnet.io

Knowledge Representation

Word2Vec

Image source: ericbern.com

Best Intro: Word2Vec Udacity Youtube

Knowledge Representation

Deep Learning

Precision Recall F-Measure Errors

People 84 73 75 <1

Dictionaries 61 57 54 8

Linear Regression 65 56 47 3

Deep Learning 62 57 49 2

Sentiment Analysis is not about maximizing F-Measure,

it’s about reducing true Errors: positive confused with negative

Theme Extraction

6.

From Words to Complex Themes

Applying Customer Feedback Analysis

Google: Sentiment by Theme

Thematic Demo

Thanks

@zelandiya

alyona@getthematic.com

getthematic.com

linkedin.com/in/medelyan