Upload
duonghanh
View
215
Download
0
Embed Size (px)
Citation preview
Words and numbers: Linguistic analyses of big survey data
Tony McEnery
ESRC Centre for Corpus Approaches to Social [email protected] | @TonyMcEnery
Our Data
• NHS Choices• Three areas focused on (comments/responses):
GP practices (14,093,437/5,596,738)Hospitals (8,605,580/4,218,888)Dentists (4,306,698/1,460,343)
• In total we have: 28,971,142/11,692,555• Too much data to be read – reading for comprehension
(200–400 wpm). Over a year (428 days or so) just to read them if you read 200 words a minute for 8 hours a day with no weekends or holiday breaks!
• So we analyse it using corpus linguistics
You are using what?
• Corpus linguistics - using computers to allow us to study large volumes of language – sometimes millions, sometimes billions of words.
• You may not have heard of the approach – but you have almost certainly benefitted from it.
• Very much a British/European innovation which spread to be used more widely.
• We have been pioneering work in this area for over 40 years.
Starting to use a corpus
• You can search for, retrieve and undertake some processing on words effectively.
• Concordance programs allow you to do this swiftly and accurately, presenting data in a way that allows it to be gisted rapidly
• You can then carry out procedures on the data which allow you to gain deeper insight into the use of language itself
• Consider the word ‘cause’
Methodology
Quantitative analysis is combined with
Qualitative analysis
• Corpus linguistics’ big advantage is frequency analysis • but for going deeper into meaning, setting, discourse, no automatic
methods are available• the human mind is needed to work on the data, but the machine
can help!
A typical process:QUANTITATIVE QUALITATIVE QUANTITATIVE
(* Especially in concordance analysis)
Breakdown of the corpus
Word count of patient
comments
Word count of NHS
responses
GPs 14,093,437 5,596,738
Hospitals 8,605,580 4,218,888
Dentists 4,306,698 1,460,343
Pharmacies 690,629 117,858
Care providers 422,133 25,032
Clinics 400,813 110,485
Opticians 179,493 41,994
Acute Trusts 159,385 63,933
Mental Health Trusts 111,557 57,284
Care Organisations 1164 0
Clinical Commissioning
Groups
253 0
Total 28,971,142 11,692,555
Some Results - keywords
an, are, call, do, get, medical, phone, prescription, see, time, when, you
dental, dentist,
ease, emergency,
explained, feel
filling, had, happy
nervous, NHS,
professional, teeth, tooth
visit, years
A&E, admitted,
after, am, caring, clinic,
consultant, had, hospital,
hours, operation, received,
surgery, team, treated,
unit, ward, were
I, !, am, appointment, care, excellent, friendly, have, helpful, me, my, patient, staff, they, told, very
always, appointments, been, good, practice, reception, receptionist, recommend, rude,
service, surgery, this
doctor, doctors, GP, not, nurse, nurses, patients, to, wait, waiting
and, experience, pain, thank, treatment, was
GPs
Hospitals Dentists
Word Frequency across all
comments
Key in which data
sets
good 59,237 GPs and Dentists
excellent 49,090 All
helpful 43,915 All
friendly 42,378 All
rude 29,335 GPs and Dentists
professional 28,104 Dentists
caring 23,719 Hospitals
happy 22,658 Dentists
We should not assume that the positive words are always used positively. For example, the phrase “not caring” occurs 128 times, while “not happy” appears 2309 times.
told, not n’t, get, to
appointmentimpossible, weeks, no, ?
rude, worst, !poor, they, do said, ", you unhelpful, then terrible, asked avoid, awful appalling, even another, was call
be phone
system, on it however
appointments frustrating that, not ?
seem seems
but
book an if or upsee
seem week
receptionist mixed is shame online hit let depends there please OK telephone
but some generally difficult
however good
sometimes overall with improved found the reception bit usually nice happy although improvement improvements pleased
helpful very have
always friendly
great staff
polite
excellent and best practice caring care been professional all highly family service efficient surgery nurses fantastic with recommend team thank years
1 2 3 4 5
Keywords for quantitative
ratings
Word Collocates with a similar meaning Collocates that act
as modifiers
Collocates which are the
target of the evaluation
good None very, really, enough,
overall, not,
generally, pretty,
service, humour(ed), food,
advice, news, doctors,
listener(s)
rude unhelpful, abrupt, arrogant,
unprofessional, dismissive, patronising,
ignorant, uncaring, disrespectful,
obnoxious, condescending, aggressive,
unfriendly, incompetent, impatient,
unsympathetic, impolite, inconsiderate,
blunt, insensitive, obstructive, sarcastic,
unpleasant
very, extremely,
incredibly,
downright, quite,
unbelievably, so,
plain, often,
bordering
staff, receptionist(s),
helpful friendly, polite, pleasant, courteous,
caring, professional, efficient, kind,
understanding, knowledgeable,
supportive, informative, cheerful,
considerate, nice, respectful,
welcoming, accommodating, reassuring,
approachable, sympathetic, lovely,
attentive, obliging
very, always,
extremely, really,
most, so, more,
incredibly, unfailingly
staff, receptionist(s),
doctors, pharmacist
Customer Service• Receptionists are particularly singled out as being rude (although it should be noted
that they are also often described as helpful and friendly)
“Bad mannered and rude Very difficult to get an appointment, very rude staff especially at the reception!”
“…their support staff at the reception desk are awful. They're so rude and seem to think it's the norm to talk down to patients. I've just seen one almost shouting at an elderly patient because she'd got the dates of her appointment wrong!”
“unfortunately the Reception staff at both locations are rude, ignorant and unhelpful - on many occasions I have witnessed patients standing around being ignored whilst the reception staff are laughing amongst themselves.”
• This initial analysis indicates that customer service and politeness tends to be a highly salient aspect of patient feedback.
Collocates - negative Collocates - Positive
waiting room packed, full, crowded,
small, busy, hot,
cramped, stuffy,
overcrowded, tiny, dirty
clean, comfortable,
pleasant, airy, spacious,
quiet
waiting time(s) long, ridiculous,
unacceptable, excessive,
horrendous, appalling,
terrible, poor, lengthy,
joke, atrocious
reasonable, acceptable,
WaitingA potentially interesting keyword is waiting (which is key for both GPs and Hospitals). This keyword tends to occur within two main constructions – waiting room and waiting time(s).
Category Collocates
Body part hernia, cataract, hip, knee, shoulder, eye, bunion,
spinal
Type major, minor, bypass, repair, small, remove
Scheduling cancelled, after, before, following, date, prior,
scheduled, pre, during, first, second, planned
Processes anaesthetic, aftercare
Outcome successful, success
Person surgeon
Operation
• I was very upset that she cancelled the operation and not postponed.
• Administratioin diabolical For the second time in less than 18 months I have had an operation cancelled at very short notice due to disgustingly poor administration.
• However, the other keywords tend to be used in positive contexts, or are merely descriptive:
• The surgeon who performed the operation was very slick and professional and seemed to take no time to cleanly remove the polyp
• After my operation, the aftercare I received from staff at the hospital was fantastic.
• Operation under general anaesthetic was performed after 2 days and I went home the same day.
Keyword Frequency Most frequent 10 evaluative collocates
staff 159,892 friendly, helpful, rude, excellent, lovely, polite, great,
professional, good, fantastic
doctor 106,777 good, excellent, great, rude, helpful, lovely, fantastic,
caring, friendly, brilliant
dentist 73,424 excellent, good, great, friendly, lovely, professional,
fantastic, brilliant, helpful, rude
nurse 41,388 lovely, friendly, excellent, rude, helpful, good,
fantastic, great, nice
receptionist 36, 491 rude, helpful, friendly, lovely, polite, unhelpful, nice,
pleasant, good, excellent
GP 67,483 excellent, good, great, helpful, fantastic, friendly,
happy, rude, caring, brilliant
consultant 14,444 excellent, fantastic, brilliant, wonderful, professional,
great, lovely, friendly, rude, good
StaffTable 6 looks at collocates of the keywords which refer to staff. I have examined the most frequent evaluative collocates for each keyword, and put the negative ones in bold.
Who is rude?
Number
of times
called
rude
%
receptionist 3,056 8.37
staff 6,166 3.85
nurse 402 0.97
doctor 853 0.79
dentist 533 0.72
consultant 65 0.45
GP 303 0.44
Positive and negative words
• What are the key drivers of positive and negative feedback (or what do the words below collocate with?)
Positive Negative
good 59,237 bad 16,945
excellent 49,090 poor 15,274
great 34,298 worst 7,627
best 25,556 worse 7,289
fantastic 15,186 terrible 6,799
brilliant 11,546 awful 6,106
wonderful 10,371 appalling 4,410
amazing 9,749 disgusting 3,246
outstanding 5,019 ridiculous 3,206
exceptional 3,387 useless 2,461
Themes of positive and negative evaluation
• We grouped the most frequent collocates of the positive and negative evaluative words into themes:
THEME Words relating to the theme
TREATMENT care, treatment, dental
SYSTEM system, appointment, time
COMMUNICATION communication
INTERPERSONAL attitude
Proportion of positive to negative evaluation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Treatment Communication Interpersonal System / Organisation
Negative
Positive
When treatment is evaluated positively
Interpersonal (42%)
Appointments (12%)
Technical competence
(12%)
Communication (10%)
Efficiency (7%)
Cleanliness (6%)
Hard-working (5%)Other (6%)
When interpersonal skills are evaluated negatively
Impolite / rude (41%)
Apathetic (9%)
Lazy (7%)
Not listening (7%)
Abrupt (7%)
Don't answer phone (6%)
Appeared impositioned (5%)
Not smiling (5%) Other (11%)
Proportion of positive to negative evaluation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Dentists Paramedics Midwives Nurses Opticians Pharmacists Doctors Receptionists
General Positive and Negative Descriptors
Positive Negative
Some negatively evaluated receptionists
• The last and final time i tried to make an appointment at this Health Centre, the receptionist laughed in my face
• After speaking to a very snooty receptionist who was very unhelpfulwe are no further to getting the splints removed.
• Receptionist interrupted talked over me on the phone, had to ask if I could finish my sentence three times.
• Upon approaching the desk I had to wait whilst the receptionist finished telling their colleague about their weekend, then when I asked my question I was given a response of "well it 's not exactly hard to work out ".
• Receptionist had no compassion whatsoever.
• My daughter is now too embarrassed to go back when she was asked personal questions about the nature of her ailment by the receptionist .
Why do receptionists do so badly?
• Encountered by largest number of people?
• Gatekeeping role has capacity to annoy – the “face” of systems they didn’t design
• Questions mistaken as nosiness
• Is social class a factor? E.g. viewed as relatively inexpert so afforded less respect
• Is gender a factor? (94% of British receptionists are female)
Keyword Frequency Most frequent 10 problematic verbs (where available)
staff 159,892 refused, failed, ignored, complained, lost, shouted, forgot,
insisted, laughed, cancelled
doctor 106,777 refused, failed, insisted, complained, ignored, laughed,
shouted, dismissed, forgot, shrugged
dentist 73,424 refused, failed, insisted, missed, cancelled, ignored, shouted,
complained, laughed, hit
nurse 41,388 refused, shouted, insisted, complained, admitted, forgot,
failed, ignored, laughed, threw
receptionist 36, 491 refused, insisted, shouted, ignored, hung up, complained,
laughed, shrugged, failed, barked
GP 67,483 refused, failed, ignored, insisted, forgot, complained, missed,
dismissed, laughed, shouted
consultant 14,444 refused, dismissed, ignored, failed, admitted, complained
What about problematic behaviour? Below are the 10 most frequently mentioned past tense verbs which are evaluated as negative behaviours for each of the staff keywords.
• The receptionist barked out questions and orders with barely looking at me
• One Doctor laughed at me when I went in crying with Acne at age 25 ...
• As soon as she woke up she had to vomit and the nurse threw a vomit bowl at her and did not even bother to help her .
• I asked what I should do if the Dr doesn't arrive and the receptionist shrugged their shoulders and stated they didn't know .
Conclusion
• Our techniques allow large bodies of textual data to be exhaustively analysed
• The exhaustive analysis can show trends clearly
• Actors, actions, and attitudes shine through
• Some results will confirm and reinforce views, others will undoubtedly surprise
• Early days – much more work to be done. Other data sets to look at as well in the future.