Validity in Psychiatry

Maj-Britt Posserud, MD, PhD

Child and Adolescent Mental Health

Haukeland University Hospital, Bergen, Norway

Validity

” validity is the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world.”

Source: Wikipedia

Reliability and Validity

© Nevit Dilmen.

Various forms of validity

• Construct validity “the degree to which a test measures what it claims, or purports, to be measuring.”

– Cronbach & Meehl (1955) » Group differences » Correlation matrices and factor analysis » Predictive value

• Content validity ”the extent to which a measure represents all facets of a given social construct”

• Criterion validity ”is a measure of how well one variable or set of variables predicts an outcome”

Validation

1. Define «gold standard» – Reference for construct validity

2. Construct your measure – content validity, face validity

3. Apply your measure – Internal validity, reliability (cronbachs alpha)

– Factor analysis - construct/content validity

4. Compare to other measures/outcomes – Measurement invariance (CFA, SEM)

– Convergent validity w gold standard

– Discriminant validity (differentiating groups)

– Prediction (criterion validity)

The validity of categories «Diagnostic categories defined by their syndromes should be regarded as valid only if they have been shown to be discrete entities with natural boundaries that separate them from other disorders» «valid—meaning “well-founded…sound…against which no objection can fairly be brought”» «At present there is little evidence that most contemporary psychiatric diagnoses are valid…. This does not mean, though, that most psychiatric diagnoses are not useful concepts. In fact, many of them are invaluable. But, because utility often varies with the context, statements about utility must always be related to context, including who is using the diagnosis, in what circumstances, and for what purposes.» «If no detectable discontinuities in symptoms are found in large tracts of the territory of psychiatric disorder, it is likely that, sooner or later, our existing typology will be abandoned and replaced by a dimensional classification»

Am J Psych, 2003. Robert Kendell, M.D., Assen Jablensky, M.D.

ADHD symptoms

Parental report in the BCS

ADHD – DSM-IV

• ADHD symptoms:

«present to a degree that is maladaptive and

inconsistent with developmental level»

«often fails to», «often loses», «often

fidgets»

–«clear evidence of clinically significant

(distress or) impairment»

–«not occur exclusively during»

–«not better accounted for by»

Where is the deviance?

Impairment

• Intrinsic requirement for diagnosis

• However, the same degree of social ineptitude will lead to varying degrees of impairment depending on context

The diagnosis is not constant!

• Although vital for diagnosis, unspecific and not related to the underlying taxon or trait

Important concepts in screening/criterion validity:

• Sensitivity • Screen positive sick/Sick

• Specificity • Screen negative healthy/Healthy

• Positive predictive value • Screen positive sick/Screen positive

• Negative predictive value • Screen negative healthy/Screen negative

• Diagnostic Odds Ratio – DOR • Correctly classified/incorrectly classified

• Reciever Operating Curves

Factor analysis

• PCA – principal component analysis – Extracts all variance, including error variance

• EFA – exploratory factor analysis

• CFA – confirmatory factor analysis – A structure is decided beforehand – data is «fitted» to model

» Measure of model fit

• SEM – structural equation modelling – Latent variables may also be independent variables

Latent traits, SEM and measurement error

Measurement invariance & item response theory

• Construct validity • A measure may measure difference thing in different

places

• In psychiatry we generally measure latent traits

Factor analysis and structural equation modelling

Item response theory

• The probability that a person with a given trait will score on a particular item

SDQ - the world´s most available questionnaire!

• African languages: • Afrikaans

• Arabic (+ D)

• Amharic

• Chichewa

• Igbo

• Somali

• Swahili

• Xhosa

• Yoruba

• Zulu

• Asian languages: • Azeri (+ D) • Bengali (+D) • Chinese (+D) • Dari • Farsi • Georgian • Gujarati • Hindi • Hmong • Indonesian • Kannada • Malayalam • Pashto • Thai • ….

SDQ in 8 countries: UK, Norway, Bangladesh, India, Yemen, Brazil, Russia

”parent SDQ symptoms scores were so high in rural Yemen that the population mean of non-disordered children was comparable to that of children with a disorder in Britain. The converse was true of the final notably anomalous population…the 26 Indian children with a disorder.”

Conclusions:

”The result was that none of these SDQ caseness indicators could be used to make meaningful estimates of prevalence across the non-British samples.” ”population-specific norms may be needed when estimating prevalence… the SDQ shows large cross-cultural reporting effects and cannot be assumed a priori to be a valid method for comparing prevalences cross nationally without recourse to population-specific norms.” ”hihglights the importance of using local cultural and linguistic knowledge when reading the DAWBA transcripts and interpreting responses to structured questions.” ” We therefore recommend that questionnaires are only used in cross-cultural comparisons when their cross-cultural equivalence has been empirically demonstrated. Doing so may require detailed diagnostic measurements that employ local and contextual knowledge in order to provide population-specific reference points for judging the performance of brief questionnaire measures.”

SDQ Congo

Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among school

children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry, 14(4), 208-215.

1 - 28% 2 – 9.6% 3 – 6.7% 4 – 5.2% 5 – 4.9%

3 .55

8 .69

13 .61

16 .69

24 .75

5 .58

7 .38

12 .64

18 .64

22 .59

2 .80

10 .80

15 .80

21 .48

25 .69

6 .74

11 .67

14 .52

19 .48

23 .70

1 .67

4 .75

9 .77

17 .68

20 .72 Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among

school children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry,

14(4), 208-215.

Caci, H., Morin, A. J., & Tran, A. (2015). Investigation of a bifactor model of the Strengths and Difficulties Questionnaire.

European child & adolescent psychiatry, 1-11.

Receiver Operating Curves AUC – Area Under the Curve

ADHD bifactor analysis

One general factor

Hyperactivity ADHD Impulsivity and inattention also load on separate domains

ICD-10 organisation