Making Public Spaces Safer
Dr B. Nasa, Prof J. BinnerDr M. A. Ferrario, Dr W. Simm, Prof J. Whittle, Dr B. Lam;
Sheffield Management SchoolLancaster University/ Brunel University
Towards Policing Perception Maps automatically extracting quantitative analysis of
qualitative survey data
Background: Derry/NI
HOW TO build trust in the police after 30+ years of Conflict (The Troubles)?
SELF POLICING & FREE DERRY: 1960s on going clashes between Protestants and Catholics. By the end of 1971, barricades prevented access to Free Derry (Bog Side Area) . No police forces allowed.
BLOODY SUNDAY: 30th January 1972, 27 civilians were shot by the British Army Parachute Regiment during a Civil Rights march. 13 died on the scene.
Derry District Policing Partnership
Derry DPP is one of the 26 DPPs in NI
• Responsibilities:• Monitoring local police performance• Giving a voice to community views on policing• Gaining the public's cooperation to prevent crime.
DDPP Household survey 2009 Confidence and Satisfaction
DDPP Household Satisfaction & Confidence Survey’09
• All-household survey to inform next year local policing plan (2010/11)
• First-ever DPP blanket consultation by Neighbourhood
• 46,000 households: 465 responses received to the survey (1% response rate)
Verbatim vs Tick-Box
• The survey contained questions with multiple choice answers and the opportunity for the respondent to expand upon their selection in free text form.
• Traditionally, this free text is difficult and time consuming to analyse by theme and sentiment.
• Automatically extracting ‘quantitative summaries’ could help identify themes which are not part of the multiple choice options and would allow comparison across results from different districts.
Making Sense of Qualitative Data
Factual vs Perception Mapping
Map of actual crime statistics
Perceptions maps
Challenge: How do people feel...
• 36% of respondents** answered ‘don’t know’ to the satisfaction question => How can the police ‘win’ the undecided?
• Understanding ‘Why’people are undecidedmay help…
SATISFACTION (Derry City Council Area)
Yes35%
No27%
Don't Know36%
No Answer2%
Yes
No
Don't Know
No Answer
** diagram and figures from DDPP survey
Data Set: Original Source
1
2
3
4
56
DDPP IDLOCALECONFIDENCE LEVELCONFIDENCE VERBATIMSATISFACTION LEVELSATISFACTION VERBATIM
123456
Extracted – Full Data Set
First Step: 449 Records in totalSecond Step: Sentence Level Split + Clean Data+ Remove Blank Verbatim = 379 Sentence level comments (Satisfaction)= 485 sentence level comment (Confidence)
Sample Data Set – 95% CL; CI 5%
• 200 comments (Satisfaction)• 221 comments (Confidence)• Manual analysis (three researchers)
• Theme, Actionability, Sentiment (TAS)
• Automated Theme Extraction – Accuracy Testing• VYV• Bayes Classifier
TOP THEMES (Satisfaction & Confidence)Approach:1. Initial Identification of themes (Cross-checked with DDPP)2. Independent Theme Review (by three researchers)3. Up to 4 themes per comments4. Calibration and Compilation of final theme List
Emerging Results:
Satisfaction Frequency % Confidence Frequency %General ImpressionGeneral Impression 6161 17.617.6 Response Time and QualityResponse Time and Quality 7070 18.418.4
Presence and Visibility 48 13.9 Police Behaviour/Political Policing 64 16.8
Police Behaviour/Political Policing 46 13.3 Presence and Visibility 61 16.0
Offence and Crime Level 40 11.6 General Impression 38 10.0
Response Time and Quality 32 9.2 Resources incl. Admin 31 8.1
Other 29 8.4 Offence and Crime Level 23 6.0
Resources incl. Admin 21 6.1 People behaviour and Attitude 21 5.5
People behaviour and Attitude 17 4.9 Other 17 4.5
Community Policing 14 4.0 Community Policing 15 3.9
Specific Events/Location 14 4.0 Specific Events/Location 15 3.9
Complexity of Situation 10 2.9 Complexity of Situation 13 3.4
Authority/Law enforcement 10 2.9 Perceived Safety 7 1.8
Perceived Safety 4 1.2 Authority/Law enforcement 6 1.6
Total – themes 346 100 Total 381 100%
THEMES BY LOCATIONSecond Step: Match Themes to NeighbourhoodsNeighbourhood analysis of Satisfaction with Policing in Derry Themes (aggregated) Foyle City
CentreCity North Foyle City
WestWaterside
RuralWaterside
UrbanTotal by ThemesFreq. %
General Impression 8 17 8 17 11 61 17.6
Presence and Visibility 2 13 9 16 8 48 13.9
Police Behaviour/political policing 4 13 7 13 9 46 13.3
Offences and Crime Level 3 14 8 11 4 40 11.6
Response Time and Quality 1 2 6 11 12 32 9.2
Other 5 3 4 10 7 29 8.4
Resources Incl. Admin 2 3 5 4 7 21 6.1
People Behaviour 3 7 2 2 3 17 4.9
Community Policing - 5 3 3 3 14 4.0
Specific Events/Location - 4 2 5 3 14 4.0
Complexity of Situation 3 2 - 4 1 10 2.9
Authority/Law Enforcement - 5 1 1 3 10 2.9
Perceived Safety - - - 4 - 4 1.2
NeighbourhoodFreq. 31 88 55 101 71 346 100.0
% 9.0 25.4 15.9 29.2 20.5 100.0
3-WAY ANALYSIS: themes v. sentiments v. location
SATISFACTION v. SENTIMENTS
Is stated level of satisfaction reflected in verbatim
Satisfaction Level
Sentiment derived from verbatim
Negative Neutral Positive Total
No 109 14 0 123
32.15% 4.13% 0% 36.28%
Don’t Know 46 40 6 9213.57% 11.8% 1.77% 27.14%
Yes 14 32 78 1244.13% 9.44% 23.01% 36.58%
Total 169 86 84 33949.85% 25.37% 24.78% 100%
Null: SL is independent of VSChi square statistics : χ2 = 211.04 (0.000 )Reject null, so SL is associated with VS
ExampleVERBATIM ANSWERS**
Do you think the PSNI are doing a good job in the Foyle area? Don’t know – PSNI take too long to respond to a reported incident.
Don’t Know - They certainly don’t appear to be able to curb the problems in my area of Culmore.
Don’t Know -Because they let the young joy riders straight out again also they know all the drug dealers in the one area still they turn a blind eye.
Automated Theme Extraction
VYV-Bayes Classifier
Accuracy of Automated VYV Theme Extraction Approach:
1. VYV automatically assigned (two) themes
2. Accuracy assessed on a 3-point Likert scale
3. Independent assessment by 2 reviewers, followed by a ‘calibration exercise’ to address differences
4. 69-72% of the themes were acceptable or accurate for both sets
5. Reviewers agreed on both themes 93-94% of the time
• Noun picked from POS = “Staff” sem tag I3.1
• Secondary sem tag S2
<w pos="NN" lemma="staff" sem="I3.1/S2">staff</w>
“Very helpful and friendly staff”
VYV tags vs Manual (Satisfaction)
VYV identified a total of 90 unique themesManual tagging identified 17 themesSome Interesting Parallels and Challenges
Unique Themes freq % rank (top 17)Places 34 9.7% 1Work and employment: Generally 14 4.0% 2Evaluation: Good 12 3.4% 3Speech acts 11 3.1% 4Belonging to a group 11 3.1% 5Difficult 11 3.1% 6People 10 2.8% 7Time: Period 10 2.8% 8Grammatical bin 9 2.6% 9Time: New and young 9 2.6% 10Location and direction 9 2.6% 11Helping 9 2.6% 12General actions / making 9 2.6% 13Existing 9 2.6% 14Vehicles and transport on land 9 2.6% 15Crime 8 2.3% 16Getting and possession 7 2.0% 17
Theme Assignment Alternative
Classifier methods can be used to classify texts:
1. The classifier is trained using manual data2. A number of exclusive classifications are defined 3. As subset of data is selected and manually classified4. This subset is used to train the classifier on the
features (in this case the words) that make up each classification
5. New comment are classified by the classifier based on features
Bayes Classifier applied to DDPP Data
• Naive Bayes Classifier was applied to “Confidence” data:• Training subset used previous 83 manually tagged
comments• classifier accuracy increases with number of training data• 20 further comments were extracted at random and
classified by the Bayes classifier to test accuracy.
• Preliminary Results:• In a manual review by 2 researchers (method described
previously) the classifier returned an “Acceptable” or “Accurate” classification for 60% of the 20 comments.
Towards Perception Maps
Interactive Map Display
• Plot the information derived from automatic analysis of survey verbatim on an interactive map
• Allows interaction with, and evaluation of the data by stakeholders and possibly public
Safer Streets
Implementation
• Design Storyboard by Information Architecture student Zoe Zhao (Brunel University)
• Development Progress using web technologies• Google maps• Web 2.0 technology including Ajax and JQuery
1. Frontpage, which contains:
i. Map of the whole area
ii. Overall impression – e.g. Themes, Number of People Reporting, etc
StoryBoard A
StoryBoard B
2. Click on the map to select the area that you want more information
3. The selected area will be highlighted
4. The information of the specific area replaces the overall impression
StoryBoard C
5. Select the information that you want to explore further
6. Place the cursor on the theme for actual figures
7. You also can rate your satisfaction level in this category
8. The sentiment charts will be updated automatically
StoryBoard D
9. Click on the themes for actual comments
10. You will see both positive and negative comments under this theme
11. You can enter your comments or respond to other people’s comments
12. You will see your comment appears on the list straightaway
DDPP Survey Data 2009
Challenges:
• Database design• Google map plots• Ajax for realtime updates• Jquery interface animation
Conclusion and Next Step
• Empirical Analysis• Test specific hypothesis such as
• whether there is more crime/offences where police are sectarian
• Is response related with police behaviour
• Automatic TAS• Train bayes classifier using larger dataset with fewer
categories
• Interactive Map• Put the plans into action
Partners & Collaborators
www.voiceyourview.com