kognit.dfki.de

Kognit – Cognitive Assistants for Dementia Patients

Daniel Sonntag German

Research Center for Artificial Intelligence

AI IUI HCI

Interdisciplinary Field and Transcommunity

https://dl.dropboxusercontent.com/u/48051165/ISMAR-2015-IUI-TUTORIAL.pdf

AI as the Basis for Multimodal Interaction in IUIs

Multimodal Multisensor Interfaces

Kognit Theory Design Aspects

Topic

• The use of AI to elders with dementia

• Intelligent assistive technology

• Intelligent cognitive assistance technology

• Design of advanced assistive technology

• While many older adults will remain healthy and productive, overall this segment of the population is subject to physical and cognitive impairment at higher rates than younger people.

• Because of the demographic chance, here will be fewer young people to help older adults cope with the challenges of aging.

• Intelligent cognitive assistance technology may enable older adults to “age in place,” that is, remain living in their homes and independently for longer periods of time.

Motivation

Kognit’s win-win effectImprove quality

of life Living a self-determined independent

life.

Save enormous amounts of

money

Provide relief and

more time for caregivers

Reduce healthcare

systemcosts

Institutionalisation has an enormous

financial cost

The change in demographics is immediately clear: older adults make up an increasingly greater proportion of

the population.

The most rapid growth will occur within a subgroup of this cohort—the so-called “oldest old,” or people over the age of 80.

Compensate for the physical and sensory deficits that may accompany aging

no computer technology

- lift chair, wheel chair- ergonomic handles

- hearing aid device- cardiac pacemaker

Advanced computer-basedtechnologies for AAL

(ambient assisted living)

- SSPI - exoskeleton

- control household appliances (using, e.g.,

head gestures)

Towards cognitive enhancement

computer technology

AI technology

- SSPI (Speech)- AI companion

Assurance of, compensation for, assessment of cognitive deficits

CIND / Dementia

sensor-motor andpsychosocial issues

cognitivedecline

Goals for Kognit

Assurance and Monitoring:

ensuring safety and well-being and

reducing caregiver burden, by tracking an

elder’s behaviour, assessment of regularities and

providing up-to-date status reports to a

caregiver.

Compensation:provide guidance to

people as they carry out their daily activities, reminding them of what they need to do, how to do it and related

this to active memory training (in

AR, MR, VR and serious games)

and proactive multimodal help (in

the field of view).

Assessment: attempt to infer how

well a person is doing—what his or

her current cognitive level of functioning

is—based on continual multimodal

observation of performance of

routine activities (in MR, VR and

speech-based serious games)

Compensation Paradox

compensation

user must be made aware of planned task/activity and must be guided

user and caregiver satisfaction- usability / utility

avoid introducing inefficiency into user activities

- usability / utility

avoid making the user overly reliant on the compensation system

request confirmation about whether an activity has been completed

successfully

Sensors for Activity Monitoring

VideoCameras

GPS

BluetoothBeacons

Eye Tracker

SpeechInput

Bio-Sensors

Domain and location model

Interaction with smart objects

Activity recognition

Activity performance

SeriousGame

Cognitive Status

Task and user model

Context Models

AI Technology

• plan generation and execution monitoring

• reasoning under uncertainty

• machine learning

• natural language processing

• intelligent user interfaces

• robotics and machine vision

• collaboration with colleagues having expertise in

• sensor-network architectures

• privacy and security, and

• human-machine interaction

• failure to eat or drink regularly, pill taking

• wandering around

• ATM: don’t give your money to strangers

• avoid stress situations or recover from them

• household chores and many more …

• https://www.linkedin.com/topic/group/cognitive-systems-institute?gid=6729452

Scenario demands

Kognit Storyboard and implementation

• Memory disorder result in loss of episodic memory in particular, which accounts for our memory of specific events and experiences that can be associated with contextual information. Towards compensating such mental disorder, our goal is to provide the user with episodic memory augmentations by using AI technologies.

• Autobiographical events (times, places, associated emotions, and other con- textual who, what, when, where, why knowledge) that can be explicitly stated constitute information fragments for which a prosthetic memory organisation would be needed.

• A major question concerns the recall of only useful information along the thought process of the individual (and not to slow it down).

• For everyday memory support, we aim to develop a system that can recognise everyday visual content that the user gazes at and construct an episodic memory database entry of the event. The episodic memory database is used to save and retrieve the user’s personal episodic memory events.

kognit.dfki.de/media

Text Recognizer “aspirin”

Databases and recognition modules

Object DB

Activity DB

Episodic Memory DB

{ id: “bread”, type: “object”,   image: {“sample1.png”, ...},  features: {“feature1.txt”, ...}, description: “bread is a food” } { id: “Takumi”, type: “person”, image: {“face1.png”,…}, features: {“feature1.txt”,…} description: “Mitarbeiter” } …

{ id: [UNIQUE ID], start: "2014/10/30/20:10:14", end: "2014/10/30/20:10:16", activity: "eat", object: "bread“ } { id: [UNIQUE ID], start: "2014/10/30/10:05:54", end: "2014/10/30/10:20:12", activity: “discuss", object: “Takumi“ } …

{ id: “eat”, level: 2,   derived_from: {“bite”,”chew”,…}   form: “have_a_meal” } { id: “have_a_meal”, level: 3, derived_from: {“eat”,”drink”,…}, form: “” } { id: “discuss”, level: 2, derived_from: {“look_at_face”,”speak”,…}, form: “meeting” } …

Kognit Cloudant Database: https://kognit-tt.cloudant.com/

Face Recognizer

Person A

BreadObject Recognizer

Object database (including faces)

Sensor Data Attention to …

Gaze

Gaze

Face Object Text

Gaze

Episodic memory event encoding model (Breakfast Scenario)

Encoded Event

Person A

Person A

Cheese

Spoon

Bread

“ingredients:…”

“take 1 pill in the morning…”

Eye tracker (with scene camera)

GPS, or other sensors Location

Living room

Interpretation of raw sensor data: e.g., object recognition, location

estimation,…

Encode the observations into an

episodic event: [Activity] -> [Object]

Activity database (created by

crowdsourcing platform:

LabelMovie)

Make a sandwich

Speak with Person A

Read Medication Instruction

Kognit Hardware Overview

Narrative Clip Pupil Labs Eye-tracker

Oculus DK 2

Anoto

SMI Eye-tracker

Space Glasses Meta Pro (3D cam)

NAO Humanoid Epson Moverio BT-200

Structure Sensor

Brother AirscouterTobii EyeX

WheelPhone

Low range 3D cam

Cybershot scene cam

Leap Motion Accu LED projector

Serious Games in VR

Conclusion• Towards constructing episodic memory event database of the user (as the basis for

compensation), we developed a method for recognition of the visual content that the user gazes at in an everyday scenario.

• Though the face recognition showed robustness, we still have to improve object recognition in natural environments.

• In future work, we will use an HMD to present the information of previous events or recognised objects to the user to further evaluate the presented technical implementation of episodic memory along the thought-process of the user.

• 2D —> 3D, deep learning, GPU