Upload
vincenzo-pallotta
View
568
Download
1
Embed Size (px)
DESCRIPTION
Paper presented at DART2011 workshop in Palermo, Italy.
Citation preview
Interac(on Mining: the new fron(er of Call Center Analy(cs
Vincenzo Pallo:a Rodolfo Delmonte Lammert Vrieling David Walker
© 2011 interAnaly(cs 1
Outline
• Call Center Analy(cs • Automa(c Argumenta(ve Analysis for Interac(on Mining
• Experiments with Call Center Data • Conclusions
© 2011 interAnaly(cs 2
CALL CENTER ANALYTICS
© 2011 interAnaly(cs 3
Call Center Analy(cs
• Call centers data represent a valuable asset for companies, but it is oOen underexploited for business purposes because: – it is highly dependent on quality of speech recogni(on technology
– it is mostly based on text-‐based content analysis. • Interac(on Mining as a viable alterna(ve: – more robust – tailored for the conversa(onal domain – slanted towards pragma&c and discourse analysis
© 2011 interAnaly(cs 4
Mainstream Call Center Analy(cs
© 2011 interAnaly(cs 5
Does not u
nveil real
insights abo
ut custom
er
sa=sfac=on
Call Center Analy(cs: metrics and KPIs
• Agent Performance Sta(s(cs: – Average Speed of Answer, Average Hold Time, Call Abandonment Rate,
A<ained Service Level, and Average Talk Time. – Quan(ta(ve measurements that can be obtained directly through ACD
(Automa(c Call Distribu(on), Switch Output and Network Usage Data.
• Peripheral Performance Data: – Cost Per Call, First-‐Call Resolu&on Rate, Customer Sa,sfac,on, Account
Reten&on, Staff Turnover, Actual vs. Budgeted Costs, and Employee Loyalty. – Quan(ta(ve, with the excep(on of Customer Sa&sfac&on that is usually
obtained through Customer Surveys. • Performance Observa(on:
– Call Quality, Accuracy and Efficiency, Adherence to Script, Communica,on E,que;e, and Corporate Image Exemplifica,on.
– Qualita=ve metrics based on analysis of recorded calls and session monitoring by a supervisor.
© 2011 interAnaly(cs 6
Four objec(ves
1. Iden(fy Customer Oriented Behaviors, – which are highly correlated to posi(ve customer ra(ngs (Rafaeli et al. 2007);
2. Iden(fy Root Cause of Problems – by looking at controversial topics and how agents are able to deal with them;
3. Iden(fy customers who need par(cular a:en(on – based on history of problema(c interac(ons;
4. Learn best prac(ces in dealing with customers – by iden(fying agents able to carry coopera(ve conversa(ons.
© 2011 interAnaly(cs 7
ARGUMENTATIVE ANALYSIS FOR INTERACTION MINING
© 2011 interAnaly(cs 8
Argumenta(ve Structure of Conversa(ons
DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a
REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a
PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+
ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b
PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a
ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}
PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}
PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+
Why was David’s proposal on microphones rejected?
© 2011 interAnaly(cs 9
Automa(c Argumenta(ve Analysis
• Based on the GETARUNS system1. • Clauses in Turns are labelled with Primi(ve Discourse Rela(ons: – statement, narra,on, adverse, result, cause, mo,va,on, explana,on, ques,on, hypothesis, elabora,on, permission, incep,on, circumstance, obliga,on, evalua,on, agreement, contrast, evidence, hypoth, seCng, prohibi,on.
• And then Turns are labelled with Argumenta(ve labels: – ACCEPT, REJECT/DISAGREE, PROPOSE/SUGGEST, EXPLAIN/JUSTIFY, REQUEST EXPLANATION/JUSTIFICATION.
1 Delmonte R., Bistrot A., Pallo:a V.,Deep Linguis(c Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora). © 2011 interAnaly(cs 10
Evalua(on
Correct Incorrect Total Found Precision
Accept 662 16 678 98%
Reject 64 18 82 78%
Propose 321 74 395 81%
Request 180 1 181 99%
Explain 580 312 892 65%
Total 1826 421 2247 81.26%
Precision: 81.26% Recall: 97.53%
ICSI corpus of mee(ngs (Janin et al., 2003)
Delmonte R., Bistrot A., Pallo:a V.,Deep Linguis(c Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).
© 2011 interAnaly(cs 11
EXPERIMENTS WITH CALL CENTER DATA
© 2011 interAnaly(cs 12
Ra(onale: implement the four objec(ves
1. Iden(fy Customer Oriented Behaviors, 2. Iden(fy Root Cause of Problems 3. Iden(fy customers who need par(cular
a:en(on 4. Learn best prac(ces in dealing with
customers
© 2011 interAnaly(cs 13
The Data
• Corpus of 213 manually transcribed conversa(ons of a help desk call center in the banking domain.
• Average of 66 turns per conversa(on. • Average of 1.6 calls per agent. • Collected for a study aimed at iden(fying customer oriented behaviors that could favor sa(sfactory interac(on with customers (Rafaeli et al. 2007).
© 2011 interAnaly(cs 14
Iden(fy Customer Oriented Behaviors
• Based on the work of Rafaeli et al. 2006. • Customer Oriented Behaviors – an(cipa(ng customers requests 22,45% – educa(ng the customer 16,91% – offering emo(onal support 21,57% – offering explana(ons / jus(fica(ons 28,57% – personaliza(on of informa(on 10,50%
© 2011 interAnaly(cs 15
Significant correla(on with argumenta(ve labels
© 2011 interAnaly(cs 16
Iden(fy Root Cause of Problems • Coopera(veness score – a measure obtained by averaging the score obtained by mapping argumenta(ve labels of each turn in the conversa(on into a [-‐5 +5] scale.
• Sen(ment Analysis module.
Argumenta=ve Categories Coopera=veness
Accept explana(on 5
Suggest 4
Propose 3
Provide opinion 2
Provide explana(on/jus(fica(on 1
Request explana(on/jus(fica(on 0
Ques(on -‐1
Raise issue -‐2
Provide nega(ve opinion -‐3
Disagree -‐4
Reject explana(on or jus(fica(on -‐5
© 2011 interAnaly(cs 17
Top 20 Controversial Topics with average coopera(veness scores and sen(ment
© 2011 interAnaly(cs 18
Coopera(veness of speakers on top discussed topics
© 2011 interAnaly(cs 19
Iden(fy problema(c customers
© 2011 interAnaly(cs 20
Select a specific customer
© 2011 interAnaly(cs 21
Visualize a selected call
© 2011 interAnaly(cs 22
CONCLUSIONS
© 2011 interAnaly(cs 23
Conclusions • New Genera(on Call Center Analy(cs requires Interac(on Mining – Call Center Qualita(ve metrics and KPIs can be only implemented with a full understanding of the customer interac(on dynamics
• Argumenta(on is pervasive in conversa(ons. – In order to recognize argumenta(ve acts, advanced Natural Language Understanding is necessary.
• Future work: – Scalability: need to process millions of call per day! – Mul(-‐language: call centers all over the world.
© 2011 interAnaly(cs 24
The Team
© 2011 interAnaly(cs 25
23
Dr. Lammert Vrieling (1968) - Chief Executive Officer‣ 15 years in both profit and not-for-profit organizations as consultant, trainer/coach and as
executive. ‣ Experience in the steel and aluminium industry, multimedia publishing and newspaper,
financial services and in the not-for-profit sector.
Prof. Dr. Rodolfo Delmonte (1946) - Chief Science Officer‣ Since 1993 head of Computational Linguistics Laboratory of the University of
Venice, Italy ‣ 30+ years experience in computational linguistics‣ From 1986 to 1992 he worked with the Department of Engineering of the
University of Parma. From 1978 to 1986 worked with the Department of Dr. Vincenzo Pallotta (1966) - Chief Technology Officer‣ 30 years in ICT‣ 10 years in R&D‣ Human-Language Technology, Digital Libraries, Artificial Intelligence,
Ubiquitous Computing, Human-Computer Interaction, Usability Engineering, Information Retrieval, Web Search Engines, Semantic Web, Computational Logics, Training and Education, e-learning.
David E. Walker (1964) - Chief Operating Officer‣ 25+ years in IT as Software engineer, developer, project manager, and architect. ‣ Senior Software Solutions Architect with extensive experience in designing, developing and
delivering enterprise solutions for payment processing, human resource, healthcare, marketing, manufacturing and scientific research environments.
www.interanaly=cs.ch …find us at