Click to edit Master text styles

Click icon to add picture

Click to edit Master title style

On the Adaptive Measurement of Suicide Risk

Robert D. Gibbons Blum-Riese Professor of Biostatistics

University of Chicago

Supported by NIMH Grants R01-MH-66302, R01-MH-100155 Dr. Gibbons founded Adaptive Testing Technologies (www.adaptive testingtechnologies.com) which

distributes mental health computerized adaptive tests.

May 2017

Classical vs. IRT Measurement

Classical Measurement Model

Classical vs. IRT Measurement IRT

Arithmetic Algebra Calculus

What is CAT?

Imagine a 1000 Item Math Test

Bi-Factor IRT Model

,)()(1

= 11221

11

1=2=

θθθθαα

θαθαγdgdgP vv

jvu

jvj

vjvjjn

j

d

v

−−

−−Φ∏∫∏∫

∞

∞−

∞

∞−

.)()()(1=),|(=ˆ11

*

2=1

1211 θθθθθθθθθθ

θθdgdgL

PE vvviv

v

d

vi

idiiiii

∫∏∫u

.)()()()ˆ(1=),|( 11*

2=

211

121 θθθθθθθθθθ

θθdgdgL

PV vvviv

v

d

vii

idiiii

− ∫∏∫u

Gibbons and Hedeker, 1992, Psychometrika Gibbons et.al., 2007, Applied Psychological Measurement

CAT

• Traditional – all subjects get all items

• CAT - Subjects get different items based on severity

• Smallest number of items for fixed precision

• Develop large item banks that completely characterize a disorder such as depression

• Select items dynamically based on responses

• Maximize precision – Minimize burden

Diagnosis and Measurement are fundamentally different things

CAD-MDD – Decision Tree

Preliminary Results • Correlations with total bank scores

• Depression r=0.95 – 12 items – 389 item bank • Anxiety r=0.94 – 12 items – 431 item bank • Mania r=0.91 – 12 items – 88 item bank

• Emergency Department U of Chicago (n=1000)

• 26% MDD positive screens (>50% confidence) • 22% MDD positive screens (>90% confidence) • 7% MDD Positive + moderate or severe CAT-DI • 3% suicide screen positive • 3-fold increase in ED visits in past year in moderate/severe • 4-fold increase in hospitalizations in past year in moderate/severe • None of these patients had a psychiatric indication

• Primary Care Spain and US Latino Samples (n=1000)

• 33% MDD positive screens (>50% confidence) • 25% MDD positive screens (>90% confidence) • 9% MDD Positive + moderate or severe CAT-DI

High Frequency Measurement

Source: Rush University Medical Center, Drs. Sepehr Sani and Joao Busnello

•Adult (English and Spanish) •Depression* •Anxiety* •Mania/Hypomania* •Suicidality* •Psychosis*** •Substance Abuse** •Functional Impairment*** •Quality of Life*** •PTSD**** •Functional status and well being (Thyroid Cancer Survivors)**

•Perinatal (English and Spanish) •Depression* •Anxiety* •Mania/Hypomania*

•Child and adolescent (child and parent ratings) Ages 7-17 – English only (at least currently) •Depression** •Anxiety** •Mania/Hypomania** • ADHD** •Conduct disorder** •Oppositional defiant disorder** •Suicidality**

*Completed, validated and available today **Completed and being validated ***Completed and awaiting validation ****Under development

What can the CAT-MH Measure?

How do Patients and Clinicians Access the CAT-MH?

• The CAT-MH is available on an instantly scalable cloud platform with an API that can be easily interfaced to any EHR.

• This platform is used to screen and measure mental health disorders in primary care patients in Spain and the U.S., in Spanish and English.

• There are both web-based and app-based versions available.

How can the CAT-MH be used in Routine Care?

• E-mail sent to the patient prior to the doctor visit.

• Test(s) administered on their phone, notebook, or computer, at home or wherever they want to take it.

• Results are then available either on our servers, or pushed to the EHR. Alerts sent via text and e-mail.

• Tests can also be interfaced to a portal like MyChart.

• For routine follow-up, the patient can down-load an app that will administer tests at any time interval.

• The doctor provides a patient his/her ID to enable app. • The app schedules tests in the patient’s calendar.

• The patient can take the tests in a kiosk in the clinic.

• The tests can be administered via telemedicine sessions.

Current Uses of the CAT-MH • Rapid screening for depression, anxiety, mania, suicide

– Primary Care – Emergency Medicine – Psychiatric clinics – Children and Adults

• Longitudinal assessments during treatment – Mental Health care – Ketamine – Deep Brain Stimulation – Identify onset of perinatal depression

• Triage for assessment of severity at intake – Prioritize wait-list based on need rather than waiting time – Identification of patients that might benefit from mobile CBT – Stepped Care Model

• Pharmaceutical – Identification of patients severe enough for RCTs – Boosting the signal to noise ratio in measurement of patient reported

outcomes

Select Users of the CAT-MH • UCLA Grand Challenge

– Screen 1.8 million to develop a 10-year Registry of 100,000 patients • University of Chicago

– Emergency Medicine – Depression and suicide risk – Integrated Primary and Behavioral Health Care

• Rush University Medical Center – Orthopedic Surgery – Does depression lead to poor outcomes?

• NorthShore University Health Systems – Perinatal depression screening and follow-up

• University of Illinois at Chicago – Department of Psychiatry – Wait list prioritization

• University of Massachusetts – Suicide risk assessment in Emergency Medicine

• University of Colorado – Depression, anxiety and suicide screening in Family Medicine

• Cook County Department of Corrections – Screen all inmates in the Cook County Jail and monitor after release

• Kings College of London – Developing evidence-based psychological treatments for depression and anxiety

• Rand – PCORI grant: primary care quality improvement for depression in LGBT minorities

• Emory University – Deep Brain Stimulation studies

• Columbia University – Ketamine RCT (depression and suicidality)

• Veteran’s Administration – PTSD and suicidality studies

The Making of a Child Suicide CAT

Robert D. Gibbons David A. Brent

and Members of the Y-CAT (Kiddie-CAT) Team

The Dataset

• Y-CAT Study (Gibbons and Brent PI’s) • Depression Domain

– 177 items – Suicide subdomain – 10 items – 801 subjects ages 7-18

Methodology

• Identify suicidal children – Fit a unidimensional IRT model to 10 suicide items – Fit a mixture of two normals to the EAP scores – Classify membership in elevated component as

suicidal

• Map remaining 167 depression items to suicide classification using logistic regression

• Select items with OR>2.5 for the bank

Methodology

• Fit a bifactor model to the identified depression items and 10 suicide items using subdomains of depression and suicide.

• Simulate CAT from subjects with complete data. • Tune the CAT based on:

– Termination SE – Information threshold – Probability of item selection – 2016 simulated CATs

• Select CAT with fewest items with r>0.93 with total score

Results Unidimensional IRT Model – Suicide items

Item Factor Loadings 0.63 I made a suicide attempt. 0.80 I thought about death or about being dead myself. 0.94 I thought life wasn't worth living. 0.83 I thought my family would be better off without me. 0.94 I thought about killing myself. 0.92 I had a plan for how I would kill myself. 0.92 I said I was going to kill myself. 0.95 I thought that I would be better off dead. 0.95 I felt everyone would be better off without me. 0.97 My life wasn't worth living.

Results Mixture Distribution

2.01.40.70.1-0.6

200

160

120

80

40

0

Freq

uenc

y

Score

74%

26%

Results Crosswalk between Depression and Suicide

OR 2.5 - I wanted to be alone. 3.0 - I felt like I didn't have any friends. 3.5 - I didn't enjoy anything at all. 4.0 - I felt lonely. 4.5 - I felt listless compared to how I used to be. 5.0 - I felt blah and low down. 6.0 - I felt worthless. 7.0 - I felt like I was losing it. 7.5 - I stopped enjoying anything.

Results Bifactor Model

Item Threshold Loading Text_______________________________________ 28 -0.47 0.65 I wanted to be alone. 14 -0.17 0.80 I felt lonely. 131 0.02 0.84 I felt blah and low down. 59 0.18 0.82 I felt worthless. 132 0.18 0.88 I felt like I was losing it. 2 0.33 0.71 I felt like I didn't have any friends. 115 0.36 0.88 I felt listless compared to how I used to be. 127 0.45 0.93 I stopped enjoying anything. 68 0.45 0.74 I thought my family would be better off without me. 20 0.57 0.82 I didn't enjoy anything at all. 174 0.70 0.86 My life wasn't worth living. 69 0.90 0.88 I thought about killing myself. 70 1.19 0.83 I had a plan for how I would kill myself. 30 1.65 0.75 I made a suicide attempt.

Results Mixture Distribution

210-1-2

30

24

18

12

6

0

Freq

uenc

y

Score

32%

52%

16%

Results Simulated CAT

• Termination SE = 0.36 • Information threshold 0.90 • Probability to select most informative item = 0.5 • Average # of items administered = 7 • Min = 2 • Max = 18 • 30 of 116 subjects terminated prior to SE=0.36 • Average SE = 0.39 • Correlation with 64 item score r=0.94

Results: Live YCAT-SS Iter Question Response Score Se Info 1 My life wasn't worth living. 1 25.5 13.6 5.4 2 I felt awful or unhappy. 1 14.7 12.0 3.8 3 I felt I couldn't do anything right. 1 11.5 11.5 1.9 4 I felt grouchy and in a bad mood. 1 7.5 11.1 1.7 5 I felt grumpy and upset. 1 4.5 11.0 1.5 6 I wanted to be alone. 1 2.4 11.0 1.1 7 I had a harder time getting started than usual. 1 0.3 10.6 0.9

Iter Question Response Score Se Info 1 My life wasn't worth living. 5 73.9 10.0 5.4 2 I said I was going to kill myself. 5 84.4 8.6 7.3 3 I felt listless compared to how I used to be. 5 89.8 7.7 5.1 4 I had a plan for how I would kill myself. 5 92.0 7.6 3.5 5 I didn't enjoy anything at all. 5 96.1 7.5 3.2 6 I thought there was nothing good for me in the future. 5 99.1 7.3 1.8 7 I no longer cared about anything. 5 100.0 7.2 1.4

Iter Question Response Score Se Info 1 My life wasn't worth living. 1 25.5 13.6 5.4 2 I felt awful or unhappy. 5 51.2 8.3 3.8 3 I felt worthless. 5 58.1 7.5 6.7 4 I felt I was no good anymore. 3 54.3 7.1 6.4 5 I felt listless compared to how I used to be. 4 66.8 4.9 6.2

Adult CAT Suicide Scale - Validation

• Validation UC and UMass EDs – n=290.

• CAT-SS: 58% no risk, 31% low risk, 11% high risk.

• CSSRS: 15% ideation (9% active, 6% intent/plan).

• 52-fold increase in clinician rated suicidal ideation on C-SSRS across range of CAT-SS.

• No risk vs. high risk (CAT-SS) vs any ideation (C-SSRS) sensitivity=1.00, specificity=0.95, kappa=0.81

• Active ideation, sens=1.00, spec=0.92, kappa=.66

• JCP in press

Relevant Publications Gibbons R.D., & Hedeker D.R. Full-information item bi-factor analysis. Psychometrika, 57, 423-436, 1992. Gibbons R.D., Bock R.D., Hedeker D., Weiss D., Segawa E., Bhaumik D.K., Kupfer D., Frank E., Grochocinski V., Stover A. Full-Information Item Bi-Factor Analysis of Graded Response Data. Applied Psychological Measurement, 31, 4-19, 2007. Gibbons R.D., Weiss D.J., Kupfer D.J., Frank E., Fagiolini A., Grochocinski V.J., Bhaumik D.K., Stover A. Bock R.D., Immekus J.C. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368, 2008. Gibbons R.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kim J.B., Kupfer D.K. The CAT-DI: A computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104-1112, 2012. Gibbons R.D., Hooker G., Finkelman M.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kupfer D.J. The CAD-MDD: A computerized adaptive diagnostic screening tool for depression. Journal of Clinical Psychiatry, 74, 669-674, 2013. Gibbons R.D., Weiss D.J., Pilkonis, P.A., Frank E., Moore T., Kim J.B., Kupfer D.J. Development of the CAT-ANX: A computerized adaptive test for anxiety. American Journal of Psychiatry, 171, 187-194, 2014. Achtyes E.D., Halstead S., Smart L., Moore T., Frank E., Kupfer D., Gibbons R.D. Validation of computerized adaptive testing in an outpatient non-academic setting. Psychiatric Services, published on-line. Beiser D., Vu, M., Gibbons, R.D. Test-retest reliability of a computerized adaptive depression test. Psychiatric Services, on-line. Gibbons R.D., Computerized adaptive diagnosis and testing of mental health disorders. Annual Review of Clinical Psychology, 12, 83-104, 2016. Kim J.J., Silver R.K., Elue R., Adams M.G., La Porte L.M., Cai L., Kim J.B., Gibbons R.D. The experience of depression, anxiety and mania among perinatal women. Archives of Women’s Mental Health, published on-line ahead of print. Gibbons R.D., Kupfer D., Frank E. Moore T., Boudreaux E. Development of a computerized adaptive suicide scale., Journal of Clinical Psychiatry, in press. Gibbons R.D., Beiser D., Boudreaux E., Kupfer DJ. Einstein, measurement and prediction. Journal of Affective Disorders, in press.