A Case Approach to Rating Events and A Case Approach to Rating Events and Difficulties in the National Comorbidity Survey 2 Difficulties in the National Comorbidity Survey 2

(aka “Down and Dirty with the Data”)(aka “Down and Dirty with the Data”)

Elaine WethingtonElaine WethingtonCornell UniversityCornell University

&&

Joyce SeridoJoyce SeridoUniversity of ArizonaUniversity of Arizona

May 20, 2005May 20, 2005

AcknowledgmentsAcknowledgments

Ronald C. Kessler (Harvard)Ronald C. Kessler (Harvard) George W. Brown (London)George W. Brown (London) William Eaton (Johns Hopkins)William Eaton (Johns Hopkins) StudentsStudents

Catherine J. TaylorCatherine J. Taylor Lauren Beckles, Karina Chapman,Sarah Lauren Beckles, Karina Chapman,Sarah

Howe, Ninfa Leal, Dhurgha Reddy, Jessica Howe, Ninfa Leal, Dhurgha Reddy, Jessica Richards, Richards,

Aims of the PresentationAims of the Presentation

Introduce a case-review approach to coding and Introduce a case-review approach to coding and rating rating conventional survey measures of life conventional survey measures of life events and difficultiesevents and difficulties Will apply method to prediction of onsets of disorderWill apply method to prediction of onsets of disorder

Why we did it:Why we did it: Useful to the life course approach Useful to the life course approach Make the most of conventional survey methods and a Make the most of conventional survey methods and a

pre-existing datasetpre-existing dataset Reduce cost of producing detailed data on stressorsReduce cost of producing detailed data on stressors

SampleSample

Re-interview in 2000-2003 of respondents Re-interview in 2000-2003 of respondents from National Co-Morbidity Survey (NCS I from National Co-Morbidity Survey (NCS I – 1990-1992)– 1990-1992) N=5006N=5006

85% retention rate from wave 185% retention rate from wave 1

Interview questions at the National Interview questions at the National Comorbidity Survey web site:Comorbidity Survey web site:

http://www.hcp.med.harvard.edu/ncs/http://www.hcp.med.harvard.edu/ncs/

Measures in the National Measures in the National Comorbidity Survey 2Comorbidity Survey 2

Approximately 200 questions about life events Approximately 200 questions about life events and difficulties -- based on NCS-1 (Kessler and difficulties -- based on NCS-1 (Kessler et al., 1994); Detroit Area Survey 1985 et al., 1994); Detroit Area Survey 1985 (Kessler et al., 1984); Structured(Kessler et al., 1984); Structured Life Event Life Event Interview (Wethington et al., 1995)Interview (Wethington et al., 1995)

Onsets of depression, anxiety disorders, IED, Onsets of depression, anxiety disorders, IED, PTSD, substance abuse (lifetime and 12 month); PTSD, substance abuse (lifetime and 12 month); self-reported disability associated with disorders self-reported disability associated with disorders

Social support, personality, mood, childhood Social support, personality, mood, childhood conditions, demographicsconditions, demographics

MethodsMethods

Constructed a one-year case history of Constructed a one-year case history of life events and difficulties (follow-up year life events and difficulties (follow-up year only)only)

SAS programming stepSAS programming step: : generated event and generated event and difficulty records from close-ended questions and difficulty records from close-ended questions and dates dates

Case review stepsCase review steps::1.1. Coded open-ended responses using narrative Coded open-ended responses using narrative

text and limited demographic characteristics text and limited demographic characteristics 2.2. Scanned entire case record and eliminated Scanned entire case record and eliminated

duplicate mentions of events/difficultiesduplicate mentions of events/difficulties3.3. Rated events and difficulties on key dimensionsRated events and difficulties on key dimensions

Dimensions Rated for First AnalysesDimensions Rated for First Analyses

Event vs. difficulty Event vs. difficulty Severity, defined as long-term threat (estimated Severity, defined as long-term threat (estimated

threat 10-14 days after the event occurrence)threat 10-14 days after the event occurrence) Measure presented today conflates estimated level of Measure presented today conflates estimated level of

severity and certainty of the ratingseverity and certainty of the rating NOT “contextual threat” ratingsNOT “contextual threat” ratings

ContentContent Focus (who the event happened to: subject, Focus (who the event happened to: subject,

other, joint) and relationshipother, joint) and relationship Loss (Lazarus; Brown & Harris)Loss (Lazarus; Brown & Harris) Danger (Brown & Harris; Dohrenwend)Danger (Brown & Harris; Dohrenwend)

ProcessProcess

Strategy – put together information from 3 pieces of Strategy – put together information from 3 pieces of paperpaper SAS program step to generate lists for each respondentSAS program step to generate lists for each respondent

Coding and rating done by 5 studentsCoding and rating done by 5 students

Reduced multiple event/difficulty records to one, when Reduced multiple event/difficulty records to one, when appropriateappropriate

Each case checked by investigatorsEach case checked by investigators

Data checked and re-checked extensivelyData checked and re-checked extensively

Total process: about 9 monthsTotal process: about 9 months

Unlike some case review and stressor rating Unlike some case review and stressor rating methods… methods… Social context (e.g. availability of support from others) Social context (e.g. availability of support from others)

NOT used in rating severityNOT used in rating severity ““Objective” details onlyObjective” details only (However, humiliation and entrapment ratings were not (However, humiliation and entrapment ratings were not

possible – dependent on knowing social context)possible – dependent on knowing social context)

ALL information preserved for future useALL information preserved for future use Other coding methods possible, e.g.Other coding methods possible, e.g.

Short-term threat Short-term threat More detailed information about focus and contentMore detailed information about focus and content

Comparison of SAS Generated and Comparison of SAS Generated and Case-Reviewed Events and DifficultiesCase-Reviewed Events and DifficultiesSource of Question SAS Reviewed

9/11 1066 300Traumas (12 mo.) 1203 483Health Screening 0 28R Illness 1044 1030Employment 2569 2322Finances 1751 1188Spouse/Partner Rel. 2050 1988Children 1428 1118Social Networks 5099 4976Other Life Events 1837 1316

Total 18,047 14,749

Key Characteristics of Reviewed and Key Characteristics of Reviewed and Rated Events and DifficultiesRated Events and Difficulties

Events 10,057 68.2%Difficulties 4,692 31.8%

Events SAS coding sufficient 8,688 86.2% Intervention necessary 1,389 13.8%

Difficulties SAS coding sufficient 3,473 74.0% Intervention necessary 1,219 26.0%

Focus of Rated Events/Difficulties and Focus of Rated Events/Difficulties and Relationship to SubjectRelationship to Subject

Relationship

Spouse ChildrenOther

/Partner

Self 5769 39.1%

Joint 3042 20.6% 65.6% 15.0% 19.4%

Other 5938 40.3% 9.4% 12.0% 78.6%

Number of Events/Difficulties Reported: Number Number of Events/Difficulties Reported: Number of Cases Reporting 0 to 12 or moreof Cases Reporting 0 to 12 or more

SAS After ReviewCount % Count %

0 574 11.5 652 13.01 825 16.5 970 18.52 776 15.5 924 15.63 708 14.1 782 11.94 600 12.0 595 7.35 444 8.9 365 5.66 324 6.5 280 3.67 239 4.8 181 2.08 159 3.2 102 2.39 105 2.1 57 1.110 73 1.5 37 .711 46 .9 33 .712 or more 133 2.6 38 .5

Severity Ratings, by MethodSeverity Ratings, by Method SAS ª After Review

Events Difficulties Events DifficultiesSeverity Count % Count % Count % Count %

Severe 450 4.2 445 11.4 573 5.7 507 10.8

Probably Severe 2682 25.0 2406 61.5 2126 21.1 2509 53.4

Possibly Severe 7592 70.8 1062 27.1 7179 71.4 1617 34.5

Not severe 0 0.0 0 0.0 179 1.8 59 1.3

ª 3409 entries generated from open-ended questions and not classifiable by SAS as either events or difficulties are excluded

““Severe” and “Probably Severe” Reports: Severe” and “Probably Severe” Reports: Percent by Sex (Case Review: Weighted)Percent by Sex (Case Review: Weighted)

Events Difficulties Severe Probably Severe Probably

Male (48.6%)* 43.2 48.0 37.9 46.8Female (51.4%) 56.8 52.0 62.1 53.2

*Proportion in sample

““Severe” and “Probably Severe” Reports: Percent Severe” and “Probably Severe” Reports: Percent by Age Groups (Case Review: Weighted)by Age Groups (Case Review: Weighted)

Events Difficulties Severe Probably Severe Probably

25 – 34 (22.6%)* 19.8 25.9 22.0 21.035 – 44 (28.9%)* 31.8 33.3 36.8 29.845 – 54 (29.2%)* 29.2 28.7 26.0 30.1Over 54 (19.3%)* 19.2 12.1 15.2 19.1

*Proportion in sample

““Severe” and “Probably Severe” Reports: Percent by Severe” and “Probably Severe” Reports: Percent by Level of Education (Case Review: Weighted)Level of Education (Case Review: Weighted)

Events Difficulties Severe Probably Severe

Probably

Less than HS (12.7%)* 22.7 13.3 14.9 17.6HS degree (30.5%)* 26.5 29.7 32.4 31.3

Some college (28.0%)* 30.8 30.4 27.2 29.0

College degree (28.8%)* 20.0 26.6 25.5 22.1

*Proportion in sample

Reliability and ValidityReliability and Validity

Inter-rater reliabilityInter-rater reliability Fall-off over 12 months Fall-off over 12 months Predictive validity (relationship to onsets)Predictive validity (relationship to onsets)

Inter-rater ReliabilityInter-rater Reliability

KappaKappa AlphaAlpha

Event vs. DifficultyEvent vs. Difficulty .95.95 .93.93

LossLoss (yes/no)(yes/no) .95.95 .97.97

Danger (yes/no)Danger (yes/no) .95.95 .97.97

SeveritySeverity .89.89 .90.90

FocusFocus .89.89 .89.89

Classification codeClassification code -------- .82.82

Comparison of Falloff in Reported Frequency of Severe and Probably Severe Events

0

100

200

300

400

500

600

700

800

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0

Months Prior to Interview Month

Nu

mb

er o

f Sev

ere

and

Pro

bab

ly S

ever

e E

ven

ts

SAS Generated Known Events

SAS Generated All Possible Events

Rated Events

Preliminary Analyses of Predictive Preliminary Analyses of Predictive ValidityValidity

Case review method cleanly distinguishes Case review method cleanly distinguishes events from difficultiesevents from difficulties

Both SAS generated method and Case Both SAS generated method and Case review method show:review method show: Severe “occurrences” in month of onset are Severe “occurrences” in month of onset are

related to onset of depressionrelated to onset of depression Preliminary findings indicate that severe Preliminary findings indicate that severe

events are related to onset of depression events are related to onset of depression within 30-60 days (after that effect decays)within 30-60 days (after that effect decays)

LimitationsLimitations

Men reported less detail in open-ended Men reported less detail in open-ended questions (affects rating)questions (affects rating)

Stigmatized behavior under-reported (e.g. jail Stigmatized behavior under-reported (e.g. jail time had to be inferred)time had to be inferred)

More complicated contextual rating schemes More complicated contextual rating schemes using trained interviewers are much better at:using trained interviewers are much better at: Dating onsets and offsets of difficultiesDating onsets and offsets of difficulties

Matching related events and difficulties to each otherMatching related events and difficulties to each other

FindingsFindings

Falloff appears to be reduced – Falloff appears to be reduced – perhaps perhaps artifactually?artifactually?

Reduces cost of using case-review methodsReduces cost of using case-review methods

• Trained coders but conventionally trained interviewersTrained coders but conventionally trained interviewers

• Takes less time to code more interviewsTakes less time to code more interviews

Method can be used in very large datasets Method can be used in very large datasets • 5006 cases rated and entered in 2 months5006 cases rated and entered in 2 months

• 4 months additional checking4 months additional checking

• Previous study using more complicated methods took 9 Previous study using more complicated methods took 9 months to interview, code, and rate 100 interviewsmonths to interview, code, and rate 100 interviews

SAS generation techniques could be SAS generation techniques could be applied to many pre-existing datasetsapplied to many pre-existing datasets But you have to live with ambiguity…But you have to live with ambiguity…

Preserves more information about events Preserves more information about events and difficulties than other case rating and difficulties than other case rating methodsmethods• Test hypotheses about different rating Test hypotheses about different rating

schemes?schemes?