Embed Size (px)
Citation preview
Getting Value Out of Chat DataWHAT TO DO WHEN YOUR DATA IS NOISY, SPARSE, AND SHORT
0
Talla
NLP for internal business use cases
Smart knowledge management
Hiring!
2
What is “Chat data?”
USER2: USER3 do you have new new cal on your Talla account already? Looks like it’s not available for me yet. Would be nice if we could also get inbox support enabled since it’s so much better than gmail. cc USER1USER3: USER2 I realized that after I typed this that I was using my personal gmail when I updated to the new changes. I looked on Talla and I didn’t see the same option to update to new calendar yet.USER4: USER2 I just enabled Inbox for our domainUSER4: new calendar is set to letting google decide when to roll it out, but it looks like we can also enable it as an option nowUSER4: I've now set that to be available as well. These may take some time to show upUSER1: USER2 its been enabled for awhile.USER1: (inbox)USER1: and the new calendar is enabled, soon as google decides you are allowed to have it.USER2: Thanks USER1 USER4
3
Things similar to chat data
Sequential interactions
Forum posts
Some email
IT ticketing system interactions
Short text
Associated with a user
Possibly directed at another user
Highly context dependent
4
Problems with chat
Increasing number of data sources
In theory contains lots of valuable information
In practice data is unlabeled
“Water, water, everywhere, but not a drop to drink.”
5
Goal: Issue detection and matching
People get help through chat platforms
Extract that data and automate the process
USER1’s interaction should help USER3!
USER1: Hi, does anyone know if we have patriot’s day off?USER2: Yeah USER1, we do.USER1: Thanks! …USER3: Hey, do we get patriot’s day off?
6
Automating knowledge delivery
Find issues or questions that people have
Match new issues to pre-existing ones
Serve the appropriate response or answer
Extracting answers is very hard
Focus on matching and search
7
Overview
Jumpstart ML: Active Learning
Topic modeling
Dimensionality Reduction and Representations
8
Find questions and analyze
Use patterns to find questions
Has ‘?’ token
Has a question word
Not too hard
Good start for finding past issues
9
Problems with extracted questions
Most questions need context to understand. e.g.:
“What is it?”
”Can I use her personal email?”
Intent varies:
Want information
Do this thing for me
Huh?
10
Only some questions make sense out of context
“Who is she?” “What is that?” “Will that fix my computer?”
Anaphora—it, that
Pronouns—He, she, etc
“What day is it?”, “Where am I?”
Answer depends on time, person asking
Requires more involved data model
11
Questions have different intents
“Performative” – Please help me? ex:
hi can you please help me reset my 2 factor authentication on salesforce?
“Informational” – What is it?
what's the pl code?
“Navigational” – How do I do this?
how do i record a vidyo meeting?
12
Can we write special case rules?
Borderline cases
is there a way to find out the size of an hbase table? – User asks “Is there (a way…)” to get directions
can anyone tell me where i find the out of stock request report? –User asks someone to give them information
Many variants
Alternative is to label data and use supervised learning
13
We want to label data, but…
Managing crowdworkers:
Expensive
Time consuming
Can’t be used unless data is safely anonymous
Will the model work afterwards?
14
Active Learning makes labeling more efficient
More value for your time
Can use with crowd workers or without
Good for chat:
Models train fast
Quick to annotate
Supervised learning with little labeled data
Annotate
Train/Predict Get data
15
How it works (roughly)
Annotate 𝐷0 ∈ 𝐷
Train your model on 𝐷0
Predict labels on remaining data (𝐷 − 𝐷0)
Choose more data, 𝐷1 ∈ 𝐷 − 𝐷0,
Choice of 𝐷1 is based on label predictions
Repeat
???
Profit!
Annotate
Train/Predict Get data
16
Where we are
Jumpstart ML: Active Learning
Topic modeling
Dimensionality Reduction and Representations
17
More to data than questions or intent
What do people talk about?
What kind of issues are common?
Are there clear lines defining topics?
Finding problem areas
Strategic thinking about what to tackle
18
Know Your Data
Read some of it (if you can)
Learn the context
Cluster and overview
19
Clustering or modeling chat topics
LDA, LSA, NMF, others
Human supervision necessary for interpretation(boo!)
Messages short, so chat is hard
Larger documents have broader topic distributions
We expect messages to be about fewer topics
20
Using LDA with Chat
𝜶 =. 𝟓 𝜶 =. 𝟏 𝜶 =. 𝟎𝟓 𝜶 = . 𝟎𝟑
know; does; link database; jermaine; running file; area; bank free; jermaine; database
did; try; work online; palace; sorry mean; try; screen user; hi; email
send; test; agent try; user; free did; ok; want client; server; user
look; able; mean user; client; error error; server; user ok; did; update
online; help; screen mean; app; does whats; agent; end mean; user; file
hi; palace; property shall; working; process client; property; user online; user; change
email; error; just emails; kelly; time online; user; update mandy; wrong; chance
user; issue; want did; ok; property palace; live; test owner; end; invoice
client; need; check ticket; whats; right run; right; check want; error; agent
owner; report; password check; chloe; duncan emails; know; link live; palace; try
21
Where we are
Jumpstart ML: Active Learning
Topic modeling
Dimensionality Reduction and Representations
22
Why do dimensionality reduction?
We want to improve our supervised learning techniques
Chat data is even more sparse than many NL datasets
Good representations can help search and similarity models
Off the shelf representations are good
Off the shelf + custom representations are better
23
Setting up methods for learning
Word2vec, NMF, even LDA
Most methods equivalent*
Chat has no clear document barriers
Methods assume either continuous context or separate documents
Using messages as contexts too sparse
24
Choosing a context
Representations are influenced by context choice
Figure out your goal
Choose context where words are associated in a way helpful for your goal
For our purposes: Words should be similar if they occur together in issues people have
25
Using a time-based context window
Window before each question
Problem statement and questions should be related
USER2: Can I email this form, or do I have to print it out?USER1: You need to drop the form off in personUSER2: OK, sure. USER1: Great.USER2: Where can I get access to the printers? …
26
Keywords are extracted from recent history
USER2: Can I email this form, or do I have to print it out?USER1: You need to drop the form off in personUSER2: OK, sure.USER1: Great.USER2: Where can I get access to the printers?…
27
Similarity from resulting representations
‘printer’
['printer', 'choice', 'fuji', 'xerox', 'settings', 'sequence', 'default', 'rollover', 'driver', 'takes', 'smaller', 'main', ]
‘issue’
['issue', 'resolved', 'helping', 'experiencing', 'companies', 'related', 'assuming', 'reported', 'double', 'site', 'saw', 'causing', 'understand', 'sorted', 'logging', 'heard’]
‘ssh’
['ssh', 'config', 'dhcp, 'ping', 'reconnect', 'jpg’, 'webconsole', 'coats', 'lab’, 'browsers', 'instances', 'bypass’]
28
Final Thoughts...
Tip of the iceberg
Understand how people interact
What information can we extract?
Can we escape our corpus?
29
Thank you everyone!
thanks
['heaps', 'great', 'perfect', 'fantastic',]
30
