Conversational Services for Multi-Agency Situational Understanding

Alun Preece (Cardiff)Dave Braines (IBM/Cardiff)

A bit of context• 2006-2016: US ARL & UK MoD funded

IBM-led £90M+ Network & Information Sciences (NIS) International Technology Alliance

• 2015: Partly based on the success of NIS ITA, Cardiff University established an interdisciplinary (soc sci + com sci) Crime and Security Research Institute (CSRI)

• 2016: ARL/MoD awarded the IBM-led $80M+ Distributed Analytics & Information Sciences (DAIS) ITA for up to 10 years

– Preece is UK Academic Technical Area Lead (TAL); Braines is UK Industry TAL

• 2020: CSRI will move into a new £60M building housing the “World’s First” Social Science Research Park (SPARK)

NATO Summit “OSCAR” Cardiff / IBM collaboration (NIS

ITA, 2014)

SPARK Building (2020)

DAIS ITA Goals

§ Derivation of situational understanding of complex situations by human users synergistically supported by machines

§ Distributed integration & exploitation of coalition data & information across heterogeneous information infrastructures

§ Dynamic adaptation of secure, resilient context-aware information systems

Advance Distributed Analytics & Information Science for Coalitions

Zero-Overhead Principle

“No feature may add training costs to the user”

“We observed that regardless of how powerful the new technology was that we deployed, our largest challenge was getting the analysts to adopt it.”DJ Patil, Building For The Enterprise — The Zero Overhead Principle, TechCrunch, 2012

Conversational Things

Envisioning of ISR data-to-decision chains in terms of ‘conversations’ among humans, devices, and services

Conversational Sensing (2014)Conversational Sensemaking (2015)Conversational Homes (2017)

Analytics services Decision makerData sources

Human-Machine Conversation (NIS ITA)Thinking and processing should be as close as possible

…so we need a language that is both thinkable and processable

Natural Language Controlled

NaturalLanguage

JavaXML

LogicProlog

Proc

essi

ng

Articulation as Language

Photographer: Sebastian Kaulitzki | Agency: Dreamstime.com

http://github.com/ce-store

Controlled English (CE)• Talking to machines in natural language is ideal but hard• CE as a compromise: “easy to read, harder to write”• Let’s bring the two together:

– Human users write NL sentences [easy to write]– Machine users convert to CE [easy to process]– Machine users respond in CE by default [easy to read]

there is a person named ‘John Smith’ that lives in Cardiff

and is a doctor.

low complexityno ambiguityITA ControlledEnglish (CE)

http://github.com/ce-store

Knowledge modelling in CEConcepts are introduced via “conceptualise” sentences:

conceptualise a ~ group ~ G.conceptualise an ~ ideology ~ I.

Relationships and properties can be added:conceptualise the group G ~ promotes ~ the ideology I and has the value T as ~ twitter account ~.

promotes is a relationship; twitter account is a property

Open Source CE Implementations

• Feature light (e.g., no CE rules)• Runs offline on edge devices or

server via NodeJS• Policies specify inter-agent comms• Open source: cenode.io

ce-storeCENode

• Feature rich• Extensible agent architecture• Java and HTTP APIs• Distributed design, in-memory DB• Open source: github.com/ce-store

ConversationalinterfaceWe defined a “speech act” protocol to enable interactions that flow between NL and CE

ask/tell

confirm

why

gist/expand

NL to CE

CE to CE

CE to NL

SHERLOCK is a crowdsourcing game, simulating tactical intelligence tasksSimpleHumanExperimentRegardingLocallyObservedCollectiveKnowledge

video: http://bit.ly/1O2jtsb

Participant taskingParticipants are given a set of ‘mugshots’ of the POIs so they can recognise them on 18 postersThey are tasked with answering 36 questions, e.g.:

What character eats oranges?Where is Giraffe?What character is wearing a red hat?What sport does Hippopotamus play?

IEEE HMS paper, May 2017

Reports results from the first SHERLOCK game (human actors):• Usability (operationalized as performance of adding facts to the

KB): despite close to zero training, 74% of the users inputted NL that was machine interpretable and addressed the assigned tasks

• Agent interaction capability (confirm-only vs confirm+ask-tell): no difference in performance

http://orca.cf.ac.uk/100131/

ICCRTS paper, Sept 2016(International Command & Control Research Symposium)

Reports results from 2nd set of runs (cartoon posters) comparing:• Online Condition: shared KB dynamically updated• Offline Condition: simulating unreliable connectivity at the edgeOffline participants outperformed online participants in information quantity, with no difference in quality

http://orca.cf.ac.uk/93425/Best paper award

Experimentation and outreachOct 2016

May

201

6

Watson@Wimbledon (IBM UK)Bringing the fans closer to the data via conversations

video: http://bit.ly/1knCLyI

• IBM Watsonfor unstructuredtext data

• Our conversational research for structured (database) data

• Realtime alerts based on stream processing

Conversational sensemaking

http://upsi.org.uk/oscar

OSCAR: Conversational Sensemaking

Primary site

Field teams

Secondary site

Social mediadata

NL

CE

Apps

KB

See: sl.dais-ita.org & nis-ita.org

Knowledge management in NIS & DAIS

Conversational places

9th International Conference on Advanced Cognitive Technologies and Applications, April 2017

(journal version in progress)

Recent experiment:• 12 participants• Simulated apartment

environment• 5 tasks (e.g, “Turn on the hall

light”)• Users are productive with zero

traininghttp://orca.cf.ac.uk/99165/Best paper award

“Alexa, ask Sherlock…”

https://flyingsparx.net/blog

Conversational City Hub

Tellability – fast data

Zero training & tooling – just chat

Transparency – ‘why?’ / logging

SPARK (early 2019)Reflection: what are the use cases?

“the world’s first Social Science Research Park”

• Co-location of academia, industry, and public sector stakeholders• Facilities include visualisation lab, secure data facility, human-information

interaction experimentation suite• Huge potential for socio/cognitive research and development

SPARK (early 2019)

Thanks to…Jon BakdashDave BrainesAndy DawsonDiyana DobrevaDan Harborne

Elliot HowellsMartin InnesTrudy LoweNick O’LearyGavin Pearson

Diego PizzocaroColin RobertsAnna ThomasWill WebberleyErin Zaroukian