To what extent can diagnosis be reliable and valid?

Actually, I felt fine before you

started diagnosing me!

Bulimia nervosa

Major depressive disorder

Post-traumatic stress disorder

Key phrases and words in titleIn title Definition

To what extent?

This is the central argument: how much is diagnosis valid and reliable, and why; and how much is it not valid and reliable, and why?

Reliable As related to this question: inter-rater reliability is when different psychiatrists or clinical psychologists agree on a diagnosis, using the same diagnostic manuals. (Using different diagnostic manuals you are testing the reliability of the manuals). Test-retest reliability is concerned with whether the same person will received the same diagnosis on two different days.

Valid As related to this question: the main issue is if people with a certain disorder are correctly diagnosed and those without the disorder are not diagnosed. Construct validity is when a diagnosis has a predictive quality and ties in with how the person behaves. The diagnostic test should be measuring what it claims to be measuring. (i.e. a test for bulimia nervosa should measure if a person has this disorder).

Diagnosis The procedure of labelling/identifying the disorder, for the purposes of treatment. Usually through a formal standardized diagnostic interview.

Reliability and Validity

Brief Outline

Intro – define the terms and introduce the argument: very difficult to diagnose psychological disorders.Content-• Inter-rater reliability• Test-retest reliability• Construct validity (especially cross-culturally)• Confirmation bias and validity• Conclusion – only able to be reliable and valid to

a limited extent

The Plan (intro1)

Introduction: SET OUT THE ISSUE

Paragraph 1

• Define reliability (inter-rater and test-retest)

• Define validity (construct validity )

• Define diagnosis – examining signs (apparent evidence) and symptoms (reported by patient) .

• Define the problem: reliable and valid diagnosing is easier with physical illnesses than with mental disorders – because many mental disorders do not have a clear cause or clear symptoms of any one disorder that are the same across patients.

The Plan (intro2)

• Particular problems associated with reliability and validity:

• Reliability of diagnosis can only be guaranteed to the extent that a classificatory system (OCD 10, DSM 5, CCMD 3) is reliable and is used by different psychiatrists and the same psychiatrist at different times, to make the same diagnosis.

• Validity of diagnosis can only be shown to the extent that to be valid a diagnosis has to be “correct”. Construct validity – the test used must identify a real pattern of symptoms, and apply a correct diagnosis, so that the patient may be treated. This is even more difficult than reliability of diagnosis

The Plan (Content1)USE EMPIRICAL EVIDENCE

• Reliability of diagnosis greatly criticised in 1970s (see Cooper (1972) and diagnosis in USA vs UK). Arguments are that it is better now, because of regular modifications to the manuals. (Give examples – multi-axial classifications, agreements between ICD-10 and DSM-5 etc.)

• Inter-rater reliability: But the classification systems merely represent current understanding and there is still variation amongst them – Nicholls et al (2000) and DSM-IV, ICD-10 and Great Ormond Street system). Inter-rater reliability on 81 children with eating disorders (as diagnosed by the GOS criteria) was the following: ICD-10 = 0.357; DSM-IV = 0.636; GOS = 0.879.

• So, two psychiatrists could barely agree, using the same criteria in the same manual. The specialised manual for children had the best inter-raterreliability. Suggests that the extent of reliability depends on the specificity of the manual.

• Not only the classification systems that determine the reliability of a diagnosis. The psychiatrist’s role is vital.

• Test-retest reliability: Seeman (2007) – test-retest reliability for schizophrenia. Initial diagnosis of schizophrenia, especially in women, was often changed as the psychiatrists got to know their patients better. The extent of the reliability of the first diagnosis is therefore limited.

The Plan (Content2)

The Plan (Content3)

• Validity of diagnosis is particularly problematic when considering construct validity. (There are differences in the definitions of the same mental disorders between the classification systems ICD-10, DSM 5 and the CCMD-3).

• See Szasz and also Laing (1960s and 1970s) for early arguments that mental illness is a “social construction”, with diagnosis relying on cultural, political, and/or social context. Labelling is a form of control of those who do not conform.

• The interpretation of the psychiatrists is also a problem for validity of diagnosis. See Jenkins-Hall and Sacco (1991) for an example of racial/ethnic bias in diagnosis. This suggests that the diagnosis is not valid, even if the classification of the disorder is.

The Plan (Content4)

• Therefore the extent of the validity of diagnosis depends on the psychiatrist/clinical psychologist being able to apply the criteria and correctly diagnose and treat the patient. As shown above (Jenkins-Hall & Sacco) this is only possible if s/he does not subjectively interpret the classifications in a biased fashion.

• Relate this to cross-cultural diagnosis as problematic - the psychiatrist may not be familiar with the symptoms presented by the patient as being relevant to a particular disorder and so may not be able to arrive at a valid diagnosis (see Marsella 2003)

• Confirmation bias can give the appearance of validity to a diagnosis when there is none: see Rosenhan (1973) – once pseudo-patients were diagnosed, every behaviour they exhibited was interpreted within the framework of the diagnosis, seeming to support its validity. It was very difficult for them to leave the mental institution by convincing the doctors that they were not suffering from a disorder.

The Plan (Conclusion)

SUMMARISE THE ANSWER TO THE QUESTION• Therefore, a diagnosis can only be reliable to the extent that the

classification system used is reliable and the interpretation of the classification by the psychiatrist or clinical psychologist is reliable.

• Similarly, a diagnosis can only be valid to the extent that the symptoms are correctly interpreted and if there is agreement between the classification systems as to which particular disorder this applies.

• Given the diversity of cultures presenting with symptoms, different definitions in different classification systems, the fact that these are interpreted in varying ways, and the presence of personal bias and confirmation bias, a diagnosis can only be reliable and valid to a very limited extent.

What did you notice?

1. Was the title referred to throughout?

2. How integrated was the critical thinking?

3. Were there markers like “therefore” “this shows that”?

4. Was there a logical argument supported by evidence?

5. Did the conclusion answer the question?