Eldercare ResearchCognitive Computing for Aging Society

Hiro TakagiSTSM, IBM Research - Tokyo

Aging Strategic Initiative lead, IBM ResearchHuman Computer Interaction area lead, IBM ResearchAcademy of Technology member

62 87

28%

27%

Dementia 15% + MCI 13%estimation by Japanese Government, 2012

of people over the age of 65 are at risk of dementia.

People over the age of 65, 2017.

70%of people want to work over the age of 70 (and forever).

Life expectancy of women in 1950 and 2015.

* numbers in Japan

2017

We need a new model for active life-long society.

©2016IBMCorporation

2003“Elderly Vision Simulator”

2007Eclipse.orgOpen source

2011Official tool for JIS (Japanese Industrial Standard)

2011Japanese Government Science Grant“IT for Active Aging Society”

2009Open Collaborative ResearchUniversity of Tokyo“Elderly Accessibility”

2013Japanese Government Pilot Grant“Remote community collaboration”

2015Sekisui House JDA

2017Japan Post Official Service “Conversation as Sensors”

GTO 2016 Demographic Shift

2015Japan Post Announcement

2015 – 2016Japan Post Pilot “Voice QA”

2005 2010 2015 2016 2017

Tokyo Research Aging Research History

Research Aging Strategic Initiative2014

KDDI “Senior Smartphone”

Government and University projects Customer projects Internal activities

Power of IT – Five Long-term Focus Areas

Social

Intelligent assist

Sensing

Accessibility

Brain

Social

Intelligent assist

Sensing

Accessibility

Brain

Use case barrier

Psychological barrier

User interfacebarrier

Price Barrier

�Small fonts, small buttons�Don’t know how to operate

�Fear, uncertainty

�Cannot imagine use cases

Barriers for Senior Citizens to Start Using Information Technologies

Target Actual(average)

Fingertip angle + Parallax

Landed on

Took off

(px)

(px)

Touch duration + Tremor

Unintentional Motion

Accessibility: Touching a target location

Gesture Analysis for Novice Elderly Assist (2014)

Pressrelease&images (Japaneseonly)IBMJapanPressRelease:http://bit.ly/1gavLuNKDDIR&DLabsPressRelease:http://bit.ly/1hK1B2KKDDICorporationPressRelease:http://bit.ly/1f5zbTx

Providedappropriateguidancetousersbasedonuser'slevelofsmartphoneoperationproficiency(skillandoperationalmisstep)byanalyzingsmartphoneoperationlogswithmathematicalanalysistechnology

Improved confidenceandsubjective operation speed

33 0 3 -- 7 0 -6 067 ,3

-- 7

6 067 ,3

2 7 - 2 1

172 17

You entered“one-hundred thousand” yen

Accessibility: Voice-over Effectiveness

Social

Intelligent assist

Sensing

Accessibility

Brain

Elderly Skill Matching System – J-scouter

Senior profiledatasets

(Free-form text)

Job / ActivityDescriptions

(Free-form text)

Find bestpersons

Find bestjobs/activities

A Prime minister initiative“Work-style transformation.”“Senior workforce” is one of focuses.

A senior worker with rich experience in legal compliance is working as an advisor in a start up company

Supported by Japan Science and Technology Agency

Senior HR company10,000 seniors registered

from TV BS11By Kyodo News

“ReimaginingLife”

“Target 5 million users.” http://www-03.ibm.com/press/us/en/pressrelease/46740.wss

2015

Japan Post Project• Pilot with 1000 elderlies completed (Oct. 2015 – Oct. 2016)• Medicine reminders, video phones, shopping assist, photo sharing, etc.

Video

Global Apple+IBM Apps for Elder Services

SupportWorker

Elderat

Home

Family /LovedOnes

Social

Intelligent assist

Sensing

Accessibility

Brain

)( )

(

-- (-

(

( ))

-

( -

-

-(

Understandambiguousquestions

Pick upappropriate

answers

Learnuser

feedback

- -

Video

Social

Intelligent assist

Sensing

Accessibility

Brain

IBM Accessibility Research

IoT, Health and Aging

Ecosystem Partners / Enriched Data

Watson IoT & Health Platform

Sensors & Devices

Client’s Data

Personality Insights & Social Interaction Data

Other Data Sources

ApplicationsData &

Platforms

0101100010001001

Cognitive Services for Elders

Empowered Living

Empowered Social

Empowered Care

KnowledgeReactor

“Infrastructure Providers”

• Voice-base natural user interface for daily watchover• Analyze life patterns, feelings, interests and issues from daily conversation (e.g. cognitive decline detection)• Share information with families and care givers• Provide support that can keep the elderly self-sufficient

Conversation as Sensors

©2014IBMCorporation

Video

IntentofEachQuestionnaire

Questionnaire Intent“What did you have last night?” Dementia (Neuro Cognitive Disease) Risk

Assessment

“Did you go out today?” Activity checking“You mentioned about gardening in diary. Are you interested in community gardening?” Recommendation of activities

”Which city do you want to travel?” Marketing“The shop, you mentioned last week, is now on sale!” Promotion“We delivered a package, but you were not there. Please contact post office" Notification (business related)

25

Emotion and Physical Condition Recognition

Voice data Features

988 acoustic features for emotion recognition.

IntensityLoudness12 MFCCPitch (F0)

Probability of voicing, F0 envelope,

8 LSF, Zero-Crossing Rate

���

Labeled voice

dataset

ML-basedRegression

Estimation

Feeling good or bad

40

0

100

Physical condition

Ex: Good night

*Each circle represents one greeting.*One participant data

Emotion

NegativePositive

Confusion FrustrationDelight Flow

Physical Condition

EmotionYorktown – Speech team

Tokyo Research

Happy

Sad

Dashboard for Family and Care Givers

27

Condition

Emotion (voice sound analysis)

Personality analysis (Watson personality insights)

Daily activities

Eating habits

Dementia questionnaire scores

���

Dashboard Visualization Examples

28

“what did you do today?” = Variety of daily activities.

practice, songs, poems, musical instruments, sessions, friends, music hall, …. House (largest), cleaning, washing, room,

study…..

Smaller variety and words are not so active. Wider variety, and words are active.

“what did you have today?” = Variety of nutritionWider variety, and healthy meals

Vegetables, salad, radish, juice, natto, miso, eggs, ….

Rice (largest), bento, sandwich, ….

Smaller variety and ready-made meals

Feature extraction of gait

• Gait speed & its variability• Step frequency• Stride time variability• Step-length & its variability• Foot swing velocity• Stance and stride time

Detection• Motor function

• Fall risks• Cognitive decline

• Episodic memory• Executive function

• Diseases• Parkinson disease• MCI• Alzheimer’s disease

Gait and Cognitive Decline

Video

Social

Intelligent assist

Sensing

Accessibility

Brain

Brain Simulation: Understanding Cognitive Deficit (2016)

Video

Power of IT – Five Long-term Focus Areas

Social

Intelligent assist

Sensing

Accessibility

Brain