Detection and Computational Analysis of Psychological Signals … · Detection and Computational Analysis of Psychological Signals (DCAPS) Daniel “Rags” Ragsdale, PhD Program

Detection and Computational Analysisof Psychological Signals (DCAPS)

Daniel “Rags” Ragsdale, PhD

Program Manager

National Initiative for Arts & Health in the Military’s Third National Summit: “Advancing Research in the Arts for Health and Well-being across the

Briefing for:

Advancing Research in the Arts for Health and Well being across the Military Continuum”

February 27, 2015

1Distribution Statement A - Approved for Public Release, Distribution Unlimited

DCAPS Goal

• Develop computationally-based, networked tools to assess and analyze psychological behavioral states in a DoD medical environment.

• Analyze verbal and nonverbal cues, online activities, and indicators of social behavior

• Employ approaches both individually and in concertEmploy approaches both individually and in concert

• Use novel sources of data: iPhone, Android, Kinect, social-networks, synthetic environments, EEGs, web cams, etc.cams, etc.

• Leverage expertise from MIT, Dartmouth, Boston University, Vanderbilt, USC, & UCLA


DCAPS: BLUF

High-level contributions:

• Early recognition of indicators of psychological distress (e g depression anxiety or PTSD)(e.g., depression, anxiety or PTSD)

• More effective triage of potentially distressed service members

• Facilitate clinician trainingg

• Share DCAPS insights and data with scientific community

• Virtual humans may reduce stress and fear of judgment

Specific Applications:

• Behavioral assessment and monitoring of military personnel, pre-during and post deployment, during, and post-deployment

• Deployed to clinician training sites

• Call Centers/HotlinesCall Centers/Hotlines


Used “Honest Signal” theory as a start point (Dr Sandy Pentland MIT)

DCAPS Approach

• Used “Honest Signal” theory as a start point (Dr. Sandy Pentland, MIT)

• Data Analyzed

• Spoken and written communications (including metadata)

• Online behaviors: email, surfing habits, games, and social networking

• Nonverbal cues such as facial expression, posture, movement patterns

• Creative content of artistic expression in art and writing (SBIR)p g ( )

• Response to stimuli obtained through sensors such as EEG, EKG, GSR

• “Products”

R lit A l i t li ti th t th “h t i l ” l i• Reality Analysis: two applications that use the “honest signals” analysis

• Clinician (TRL 9) and Mobile (TRL 6)

• MINAT: text and voice analytics tools for early detection of PTSD, depression, and suicidality (TRL 6)

• SimSensei : an avatar-based tool to perform “honest signal” analysis that examines a combination of nonverbal cues using Kinect technologies (TRL 4)


REALITY ANALYSIS

Mobile

Clinician


Reality Analysis Mobile: Overall Description

• Goal: Using data from mobile phone usage provide individuals (and clinicians) by with a high-level assessment of an individual’s mental health state

d d l h h h d• Individuals have the option to share data

• Data analyzed:

• Activity / location

• Sleep patterns

• Audio Diary content

• Call and text metadata• Call and text metadata

• Call and text content (when used with MINAT)

• Analysis produced are qualitative assessments in four areas:

• Travel, Social, Sleep, Mood


Reality Analysis Mobile: Results

• Participants’ behavioral and voice data were passively gathered then assessed using predictive models

• Assessments were completed using data from clinical and non-clinical field trials*

• Clinical trial• Clinical trial

• 95 participants over a 3-month period, included weekly audio diary entries • Trial included a 30K weekly survey questions and over 51M data points of bio-

behavioral markers

• Protocol adherence was 96%

• Non-Clinical trial • Two focus groups, totaling 21 participants• Participants were comfortable with the level of privacy of the mobile app

• Participants were also comfortable sharing their mental health status

• Gather data on adoption, usability, and privacy concerns

• The predictive models produced 0.6 to 0.9 AUC (ROC curves) for a variety of symptoms [1]

7

[1] Feast, J. (2014 , February). Cogito & DCAPS, Research project review presented at PM Site Review Meeting, Cambridge, MA.

Distribution Statement A - Approved for Public Release, Distribution Unlimited

Reality Analysis Mobile: Predictive Model Results[1]

S tSymptoms

88

[1] Feast, J. (2014 , February). Cogito & DCAPS, Research project review presented at PM Site Review Meeting, Cambridge, MA.


Reality Analysis Mobile: Privacy Ratings [1]

9

[1] Place, S. (2014)Cogito Health DCAPS Final Report, Cambridge, MA.


Reality Analysis Mobile: Data Shared with Other Entities/Groups [1]

1010

[1] Place, S. (2014)Cogito Health DCAPS Final Report, Cambridge, MA.


Reality Analysis Clinician: Overall Description

• Goal: Analyze verbal interactions between clinician and patient to infer:

1 2

• Degree of engagement

• Empathy,

• Active listing

3

g

• Graphical representations:

• Tone maps

C l b l b

4

Di l Di l• Conversational balance bar

• Conversational flow gauge

• Speech pattern timeline

Dialog Display: (1) Tone Maps plots the distribution of

dynamic variation and speaking rate(2) Conversational Balance Bar indicates

th l ti ti i ti f h k• Supports both real-time and

post analysis

the relative participation of each speaker(3) Conversational Flow Gauge measures

of the fluidity of the conversation(4) Speech Pattern Timeline captures the

tt f h d i th ll

11

pattern of speech during the call.


Reality Analysis Clinician: Real-time Interface [1]

Clinician Desktop

1212

[1] Rizzo, A., Morency, L.P., and Gratch J. (December 2013) Detection and Computational Analysis of Psychological Signals: TeleCoach Final Report, Final Technical Report, Playa Vista, CA.


MINATMINAT(Medical Informatics and Analytics Toolkit)


MINAT: Overall Description

• Goal: Assign a patient’s communication into zero, one, or more “codes” that are potential indicators for PTSD, depression, & suicidality

A i t ith t i• Assist with triage

• Communication with the patient performed using transcribed audio and written text

• Analysis produced is zero, one, or more “indicators”

• Categories of indicators:

• Stress Exposure• Affect• Behavior• Cognitive state• Degrees of Impairment

• Runs as a deployable web servicep y


MINAT: Indicator Codes

• 70 indicator codes in five categories

• Categories of indicator codes

• Behaviors• Behaviors

• Stress Exposure

• Affect

• Cognitive states• Cognitive states

• Degrees of Impairment

DSM IV i i d• DSM-IV inspired

• Criteria, symptoms, indicators

COLING 2012

• Indicator codes continually refined with experience

• Reduces ambiguity

• Increases inter-annotator agreement

• Supports machine learning15Distribution Statement A - Approved for Public Release, Distribution Unlimited

MINAT: Indicator Detection


SIMSENSEI

MultiSense / TeleCoach


SimSensei: Overall Description

• Goal: Create an avatar-based tool that senses nonverbal cues that are potential indicators of mental distress

• Communicates with an individual via conversationalCommunicates with an individual via conversational

interaction

• Data analyzed:y

• Body movement

• Gestures & other physical movements

• Facial displays

• Voice

• Response latency

• Real-time analysis produced: SimSensei: Ellie

18

• Horizontal gaze, smile level, voice, attention, body movement


SimSensei / Telecoach Results

• Goal: Assess the clinical states of interest through a multimodal analysis of a patient’s verbal and nonverbal communication

• Communication with the patient is with clinician through shared video• Communication with the patient is with clinician, through shared video

• Training

• 300+ 20-minute interviews with Ellie

• Participants’ behavioral profiles were indicative of PTSD and depression

• Demonstrated potential as an alternative clinician support tool


MultiSense Overlay: Potential Real-Time Display

20

*Person in image is a veteran participant, who has consented to sharing his image.


Psychological Distress Indicators (IEEE FG 2013, Interspeech 2013, MMVR 2013)

Vertical eye gaze Smile intensityJoy – Facial expr. Sad – Facial expr.

Distress

Hand self-adaptor Legs fidgeting

Distress No-distressNo-distress

Voice energy std. Voice quality

Distress Distress No-distressNo-distressDistress

AssessmentInterviewCorpusCorpus (DAIC)

Distress No-distress Distress Distress No-distressNo-distressDistress No-distress


TeleCoach: Prediction Results

• Individuals are able to incorporate the TeleCoach indicators into their assessments of a patient’s mental distress.

• Specifically, providing individuals with TeleCoach indicators cues improved the accuracy of their assessments

Without indicators With indicators

Accuracy of depression ß = 0.59 ß = 0.72y pratings

Accuracy of PTSD ratings ß = 0.17 ß = 0.43

ß represents the size of the effect, or in this case, degree of accuracy, in regression analyses. ß ranges from 0.0, which would be no accuracy, to 1.0, which represents perfect accuracy.


DCAPS: Bibliography of refereed published papers

• Jon Gratch, Louis-Philippe Morency, Stefan Scherer, Giota Stratou, Jill Boberg, Sebastian Koenig, Todd Adamson and Albert Rizzo. “User-State Sensing for Virtual Health Agents and TeleHealth Applications,” the NextMed/MMVR20 conference, San Diego, CA, February 20, 2013.

• Stefan Scherer, Giota Stratou, Marwa Mahmoud, Jill Boberg, Jonathan Gratch, Albert (Skip) Rizzo, Louis-Philippe Morency, “Automatic Behavior Descriptors for Psychological Disorder Analysis”, 10th IEEE International Conference on Automatic Face and Gesture Recognition, April 2013.

• Sunghyun Park and Louis-Philippe Morency, “Crowdsourcing Micro-Level Multimedia Annotations: The Challenges of Evaluation and Interface,” ACM Multimedia Workshop on Multimedia Crowdsourcing, 2012.

St f S h St M ll Gi t St t Y X F b i i M bi i Al i E Alb t• Stefan Scherer, Stacy Marsella, Giota Stratou, Yuyu Xu, Fabrizio Morbini, Alesia Egan, Albert (Skip) Rizzo and Louis-Philippe Morency, “Perception Markup Language: Towards a Standardized Representation of Perceived Nonverbal Behaviors,” Intelligent Virtual Agent Conference (IVA), 2012.

• E. Suma, B. Lange, A. Rizzo, D. Krum, and M. Bolas. “FAAST-R: Defining a core mechanic for designing gestural interfaces.” In The 3rd Dimension of CHI: Touching and Designing 3D User Interfaces, 2012. Accepted for publication.

• Fabrizio Morbini, Eric Forbell, David DeVault, Kenji Sagae, David Traum and Albert Rizzo. “A ab o o b , c o be , a d e au t, e j Sagae, a d au a d be t oMixed-Initiative Conversational Dialogue System for Healthcare”. Demonstration, in Proceedings of the SIGDIAL 2012 Conference.


DCAPS: Bibliography of refereed published papers (cont.)

• N. Burba, M. Bolas, D. Krum, and E. Suma. “Unobtrusive measurement of subtle nonverbal behaviors with the Microsoft Kinect.” In IEEE VR Workshop on Ambient Information Technologies, pages 10–13, 2012.

• Saleem S ; Prasad R Vitaladevuni S ; Pacula M ; Crystal M; Marx B ; Sloan D ;• Saleem, S.; Prasad, R., Vitaladevuni, S.; Pacula, M.; Crystal, M; Marx, B.; Sloan, D.; Vasterling, D.; Speroff, T.: “Automatic Detection of Psychological Distress Indicators from Online Forum Posts”, Proceedings of the 2012 International Conference on Computational Linguistics (COLING 2012), Mumbai, India.

A th k i h S V b A N P d R "M d l b d t i f t f ti• Ananthakrishnan, S.; Vembu, A.N.; Prasad, R., "Model-based parametric features for emotion recognition from speech," Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop, vol., no., pp.529,534, 11-15 Dec. 2011.

• Rozgic, V.; Ananthakrishnan, S.; Saleem, S.; Kumar, R.; Prasad, R., "Ensemble of SVM trees gfor multimodal emotion recognition," Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, vol., no., pp.1,4, 3-6 Dec. 2012.

• Saleem, S.; Pacula, M.; Chasin, R.; Kumar, R.; Prasad, R.; Crystal, M.; Marx, B.; Sloan, D.; Vasterling, J.; Speroff, T., "Automatic detection of psychological distress indicators in onlineVasterling, J.; Speroff, T., Automatic detection of psychological distress indicators in online forum posts," Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, vol., no., pp.1,4, 3-6 Dec. 2012.

• Rozgic, V.; Ananthakrishnan, S.; Saleem, S.; Kumar, R.; Vembu, A.N.; Prasad, R. "Emotion Recognition using Acoustic and Lexical Features " Proceedings of Interspeech Portland ORRecognition using Acoustic and Lexical Features, Proceedings of Interspeech, Portland, OR 2012.


www.darpa.mil


Back upsBack-ups


Technology Readiness Levels

• TRL 1: Basic principles observed and reported

• TRL 2: Technology concept and/or application formulated

TRL 3: Analytical and experimental critical function and/or characteristic proof• TRL 3: Analytical and experimental critical function and/or characteristic proof of concept

• TRL 4: Component and/or breadboard validation in a laboratory environment

• TRL 5: Component and/or breadboard validation in a relevant environment

• TRL 6: System/subsystem model or prototype demonstration in a relevant environment

• TRL 7: System prototype demonstration in an operational environment

• TRL 8: Actual system completed and qualified through test and demonstrationdemonstration

• TRL 9: Actual system proven through successful mission operations.


Triage Assessment (VA Suicide Hotline), ICASSP 2014

We can reliably detect severe distress in hotline conversations: 80% recall in most severe cases (TR1) with only 19% False Positive Rate

1

TR3: No Referral1

TR2: Refer to Clinician1

TR1: Imminent Danger

0.7

0.8

0.9

0.7

0.8

0.9

0.7

0.8

0.9

0.3

0.4

0.5

0.6

TPR

0.3

0.4

0.5

0.6

TPR

0.3

0.4

0.5

0.6

TPR

AUC: 0 81 AUC: 0 69 AUC: 0 90

0

0.1

0.2

0 0.5 1

0

0.1

0.2

0 0.5 1

0

0.1

0.2

0 0.5 1

AUC: 0.81 AUC: 0.69 AUC: 0.90

FPRFPRFPR


Stress Exposure Labels


Affect Labels


Behavior Labels


Cognitive Labels


Domains (Degrees) of Impairment


Documents

Detection and Computational Analysis of Psychological Signals … · Detection and Computational Analysis of Psychological Signals (DCAPS) Daniel “Rags” Ragsdale, PhD Program