AN EVALUATION OF THE ‘SHORT-TERM ASSESSMENT OF RISK AND TREATABILITY’ (START)

IN KENT FORENSIC SECURE SERVICES

Rachel QuinnGrace Wright

“Risk management should be based on a plan to reduce the risk of harm occurring and increase the potential for a positive

outcome”.

(Department of Health, 2007; Best Practice in Managing Risk, pp. 14).

“Intended to assist mental health professionals in addressing the needs of mentally and personality disordered clients in a more complete fashion than

has been attempted in previously published structured professional guidelines… [And] should assist in treatment and daily management”

(Webster et al., 2004, p.4)

• The START is a Structured Professional Judgement (SPJ) Tool i.e. provides a framework for clinicians to effectively assess and manage risk.

• It is different from the other SPJs as it:– Uses entirely DYNAMIC Variables– Focuses of the individual’s STRENGTHS, and not just their

vulnerabilities.

• Main focus of previous research literature on the START is on PREDICTIVE VALIDITY and uses only Canadian and Norwegian samples.

• However the START is routinely used in UK forensic mental health populations and is recommended by the Department of Health (DOH, 2007).

An Evaluation of the START in a UK Forensic Secure Unit

Method• Trevor Gibbens Unit, Kent, UK:

– 65 Bed, Medium Secure– Males and Females– 5 Wards: ICU Acute Sub-Acute Rehabilitation Pre-Discharge

• File-review:– Completed START assessments over 18 months (665 STARTs derived from n=80)– All aversive incidents reports (n=664): START Incident Forms / IRIS Incident Forms

• START Measures:– 22 Strength scores 1 total Strength Score (0-44)– 22 Vulnerability scores 1 total Vulnerability Score (0-44)– 14 Risk Ratings 1 Mean Risk Score (0-2)

START ITEM (x22)

STRENGTHS: VULNERABILITIES:

2 Maximally

Present

1Moderately

Present

0 Minimally

Present

0 Minimally

Present

1Moderately

Present

2 Maximally

Present

High (2)

Moderate (1)

Low (0)

START Risk Area (x14)

Trevor Gibbens Unit

• Construct Validity

• Predictive Validity

• Discriminative Validity

• Concurrent Validity

• Gender differences

• Cost-analysis

What did we Evaluate?

(Do START scores reflect the construct of risk? i.e. aversive incidents)

(Do START scores predict risk behaviours? i.e. aversive incidents)

(Can START scores discriminate between Mentally Disordered Offenders (MDOs) at different stages of the forensic care pathway?)

(Are START scores related to other SPJ and outcomes measures routinely used within the TGU?)

(Are there differences in START scores between males& females?)

(Is the START a cost-effective Tool?)

Construct Validity(Do START scores reflect the construct of risk? i.e. aversive incidents)

Conclusion: START scores are significantly correlated with aversive incidents and so reflect the construct of risk.

Total Incidents

START IRIS

START Strength Scores - 0.36** - 0.26*

START Vulnerability Scores 0.38** 0.24*

START Risk Ratings 0.45** 0.31*** Correlation is significant at the .05 level (2-tailed)** Correlation is significant at the .01 level (2-tailed)

Predictive Validity(Do START scores predict risk behaviours? i.e. aversive incidents)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1M 3M 6M

Time

AUC

START_SSTART_VSTART_RRLarge Effect Margin (AUC = 0.71 and above)Mdium Effect Margin (AUC = 0.64 - 0.70)Small Effect Margin (AUC = 0.56 - 0.63)Conclusion: START scores were the most predictive of total aversive incidents at 1

month, which decreases over time to small/medium effects by 6 months. Additionally, for both Harm to Others and Harm to Self incidents, the START is highly predictive throughout the 6

months.

**

*

*

* Correlation is significant at the .05 level (2-tailed)** Correlation is significant at the .01 level (2-tailed)

Discriminative Validity (Can START scores discriminate between MDOs at different stages of the forensic care

pathway?)

0

5

10

15

20

25

30

35

ICU Acute Sub-Acute Rehabilitation Pre-Discharge

Male Wards

Mea

n ST

ART S

core

s

START_S

START_V

0

0.1

0.2

0.3

0.4

0.5

0.6

ICU Acute Sub-Acute Rehabilitation Pre-Discharge

Male Wards

Mea

n ST

ART

Scor

es

START_RR

0

5

10

15

20

25

30

35

ICU Acute Sub-Acute Rehabilitation Pre-Discharge

Male Wards

Mea

n ST

ART

Scor

es

START_S

START_V

0

0.1

0.2

0.3

0.4

0.5

0.6

ICU Acute Sub-Acute Rehabilitation Pre-Discharge

Male Wards

Mean

STAR

T Scor

es

START_RR

0

5

10

15

20

25

Acute and Rehabilitation Pre-Discharge

Female Wards

Mean

STAR

T Sco

res

START_SSTART_V

Discriminative Validity(Can START scores discriminate between MDOs at different stages of the forensic care

pathway?)

Conclusion: START scores can discriminate between offenders on different wards.

Concurrent Validity(Are START scores related to other SPJ and outcomes measures routinely used within

the TGU?)

Total Incidents

HCR-20 HoNOS Secure

START Strength Scores - 0.58** - 0.78**

START Vulnerability Scores 0.46** 0.73**

START Risk Ratings 0.49** 0.65*** Correlation is significant at the .05 level (2-tailed)

** Correlation is significant at the .01 level (2-tailed)

Conclusion: START scores are related to the HCR-20 scales and HoNOS Secure Scales.

Gender Differences(Are there differences in START scores between males & females?)

Conclusion: Female MDOs are scored as having significantly fewer strengths and higher Risk Ratings compared to males.

** *

* Correlation is significant at the .05 level (2-tailed)** Correlation is significant at the .01 level (2-tailed)

* Correlation is significant at the .05 level (2-tailed)** Correlation is significant at the .01 level (2-tailed)

Cost Analysis(Is the START a cost-effective tool?)

Conclusion: The cost of and the time taken to complete a START assessment significantly decreases by the 3rd START. By the 9th START assessment the START plateaus at

≈£15 and ≈7 minutes.

**

**

SUMMARY...• Preliminary evidence for the validity of the START in a UK forensic mental

health population, with both male and female MDOs – first study of its kind.

– Predictive of aversive incidents up to 6 months, best at 1-3 months.– Able to discriminate between MDOs at different stages of their care pathway.– Is concurrent with other well established risk assessment tools.– Highlights possible gender differences.– Cost-effective form of short-term risk assessment.– Supports importance of considering strengths as well as risks.

• This matches that of the recommendations by the Department of Health (2007) and American Psychological Association (2006).

• LIMITATIONS – relatively small sample (particularly females), issues of underreporting incidents and crude measure of costs used.

Predictive Validity of the START with Intellectually Disabled (ID) Offenders

Risk Assessment in ID offenders

• ID offenders largely ignored in the literature generally.

• Of the limited research done, SPJs used with ID offenders have been found to be good predictors of future offending (Grey et al., 2007; Lindsay et al., 2008), as has dynamic risk factors (Lindsay et al., 2008).

• No research has ever been completed on the validity of the START with ID offenders.

Aims• Address the lack of research on assessment of dynamic risk

factors in ID offenders.

• Examine the predictive validity of the START with a sample of ID offenders in a low secure hospital.

• The role of strengths and protective factors in the prediction of violence will also be explored.

Method• The Tarentfort Centre, Dartford, UK:

– 20 bed, Low Secure– Adult male inpatients – 2 Wards: Acute Rehabilitation

• File-Review:– Completed START assessments over a 3 year period (157 STARTs derived from n=28)– All aversive incidents reports: START Incident Forms / IRIS Incident Forms

• Sample: Diagnosis N (%)

Affective disorders 3.57

Autistic Spectrum Conditions 17.86

Borderline ID 14.29

Mild ID 42.86

Moderate ID 3.57

Unspecified 17.86

Organic/ genetic disorders 7.14

Personality disorder 10.71

START INCIDENT SEVERITY SCALES

DATE: PATIENT NAME:

GROUP 1: Verbal AggressionAM PM Night

○ ○ ○ 1. Makes loud noises, shouts angrily or a argues with others.

○ ○ ○ 2. Yells mild personal insults (e.g. Y o “You’re stupid”).

○ ○ ○ 3. Curses viciously, uses foul language in a anger, uses racist abuse, makes m o moderate threats to self/others

○ ○ ○ 4. Makes clear threats towards others ( (e.g. "I'm going to kill you”) or r e q requests help to control self.

DESCRIPTION OF INCIDENTS IN THIS 24 HOUR PERIOD (Time/place & activity/staff & patients involved/what happened)

NIL RETURN (tick if no incident recorded): [ ]

GROUP 3: Physical Aggression Against Others

AM PM Night

○ ○ ○ 11. Attempts to/or succeeds in keeping o potential or actual weapons.

○ ○ ○ 12. Makes threatening gesture, swings at p people, grabs at clothes.

○ ○ ○ 13. Strikes, kicks, punches, or pulls hair ( (without injury to others).

○ ○ ○ 14. Causes mild-moderate physical injury (e.g. r bruises, sprain, welts)

○ ○ ○ 15. Causes severe physical injury (e.g. r broken bones, deep lacerations)

○ ○ ○ 16. Uses weapons or uses objects as w a weapons.

DESCRIPTION OF INCIDENTS IN THIS 24 HOUR PERIOD(Time/place & activity/staff & patients involved/what happened)

START Incident Forms

Sign all entries with printed name and position.

Category of behaviour Number of incidents recorded

Verbal Aggression 1299

Property Damage/Theft 236

Physical Aggression against others 254

Self Harm 71

Suicide 35

Unauthorised Absence 28

Substance Misuse 12

Self Neglect 52

Being Victimized 92

Sexual Inappropriateness 412

Stalking/Intimidation 116

Non-compliance 283

Vulnerability 132

Fire 8

Strengths 586

Aversive Incidents:

Results

Figure 1: Graph to show ROC curve for the relationship between START risk and strength scores and incidents of physical aggression

(AUC = .710; p<.001)

Figure 2: Graph to show ROC curve for the relationship between START risk and strength scores and incidents of property damage

(AUC = .730; p<.001)

Results

Conclusion: START risk scores had a significant high predictive accuracy for ‘Physical aggression to others’ and ‘Property damage/Theft’ incidents

Results (cont.)• START risk scores had a significant medium predictive accuracy for

incidents of:– Verbal aggression over a 30 day period (AUC = .664; 95% CI = 569 - .760; p<.001).

– Suicide/Suicidal Ideation over a 90 day period (AUC = .665; CI= 536 - .773 p<0.05).   

– Self harm over a 30 day period (AUC = .692; CI = .566 - .817 p<0.05),

– Stalking and Intimidation to others over a 30 day period and 90 day period (AUC = .679; CI = .582 - .777; p<.01; AUC = .660; CI = .573 - .747; p<.001).

• Interestingly, START strength scores were also predictive of Harm to Others - Overt Aggression across 90 days (AUC = .716; CI =.596-.836; p<.001).

Summary• The first study to investigate the validity of the START

in a ID sample.– Significant high predictive validity for START risk scores on incidents of physical

aggression to others and property damage – The START predicts more powerfully across shorter time periods (30 days).– Explanations for strength scores predicting harm to others were explored, such

as variation in staff judgements on what constitutes a strength, and pathology of the patient.

• LIMITATIONS - relatively small sample (only males, low secure), issues of underreporting incidents and lack of inter-rater reliability

• IMPLICATIONS - Clinicians could use the START to identify those at highest risk of aggression and manage this appropriately.

Thank you for Listening.

Any Questions?

• For further information, please contact:

- Rachel Quinn (Assistant Psychologist) rachel.quinn@kmpt.nhs.uk- Grace Wright (Forensic Psychologist in training) grace.wright@kmpt.nhs.uk

NB the ID paper will be published in Vol 16(1) of the British Journal of Forensic Practice , February 2014.

ReferencesAndrews, D. & Bonta, J. (1995). The Level of Service Inventory – Revised (LSI-R). Toronto: Multi-Health

Systems, Inc. Banks, B., Charleston, S., Grossi, T. & Mank, D. (2001). Workplace supports, job performance and

integration outcomes for people with psychiatric disabilities. Psychiatric Rehabilitation Journal, 24, 389-396.

Department of Health (2001). Valuing People: A New Strategy for Learning Disability for the 21st Century. Department of Health (2007). Best Practice in Managing Risk: Principles and evidence for best practice in

the assessment and management of risk to self and others in mental health services. Estroff, A. & Zimmer, C. (1994). Social networks, social support, and violence among persons with severe,

persistent mental illness. In J. Monahan, & H. Steadman (Eds). Violence and mental disorder: Developments in risk assessment (pp.259-295). Chicago: University of Chicago Press.

Felce, D. (2004). Can person-centred planning fulfill a strategic planning role? Comments on Mansell & Beadke-Brown. Journal of Applied Research in Intellectual Disabilities, 17, 27-30.

Ferguson, C. J.,Averill, P. M., Rhoades, H., Rocha, D., Gruber, N. P., & Gummattira, P. (2005). Social isolation, impulsivity, and depression as predictors of aggression in psychiatric inpatient population. Psychiatric Quarterly, 76, 123-137.

Gray, N., Fitzgerald, S., Taylor, J., MacCulloch, M. & Snowden, R. (2007). Predicting future reconviction with intellectual disabilities: The predictive efficacy of VRAG, PCL-SV, and the HCR-20. Psychological Assessment. 19(4), 474-479.

Gray, N. S., Snowden, R. J., & MacCulloch, S. (2004). Relative efficacy of criminological, clinical, and personality measures of future risk of offending in mentally disordered offenders: A comparative study of HCR-20, PCL:SV, and OGRS. Journal of Consulting & Clinical Psychology, 72, 523-530.

Hare, R. (2003). Hare Psychopathy Checklist – Revised: 2nd Ed. Toronto, ON: Multi-Health Systems. Hart, S. D., Kropp, P. R., & Laws, D. (2003). The Risk for Sexual Violence Protocol. Vancouver, Canada:

Mental Health, Law, and Policy Institute, Simon Fraser University.Hillbrand, M. (2001). Homicide-suicide and other forms of co-occurring aggression against self and against

others. Professional Psychology: Research & Practice, 32, 626-635.

Hodgins, S. (2002). Research priorities in forensic mental health. International Journal of Forensic Mental Health, 1, 7-23.

Lindsay, W. & Beail, N. (2004). Risk assessment: actuarial prediction and clinical judgement of offending incidents and behaviour for intellectual disability services. Journal of Applied Research in Intellectual Disabilities, 17, 229-234.

Lindsay, W., Elliot, S. & Astell, A. (2004). Predictors of sexual offence recidivism in offenders with intellectual disability. Journal of applied Research in Intellectual Disabilities, 17, 229-305.

Lindsay, W., Hogue, T., Taylor, J., Steptoe, L., Mooney, P., O’Brien, G., Johnston, S., Smith, A. (2008). Risk assessment in offenders with intellectual disability. International Journal of Offender Therapy and Comparative Criminology, 52 (1), 90-111.

Morrissey, C., Hogue, T., Mooney, P., Lindsay, W., Steptoe, L., Taylor, J. & Johnston, S. (2005). Applicability, reliability and validity of the psychopathy checklist-revised in offenders with intellectual disabilities: some initial findings. International Journal of Forensic Mental Health, 4(2), 207-220.

Nicholls, T., Brink, J., Desmarasis, S., Webster, C., Martin, M. (2006). The short term assessment of risk and treatability (START): a prospective validation study. Assessment, 13(3), 313-327.

Parley, F. (2001). Person-centred outcomes: are outcomes improved where a person-centred care model is used? Journal of Learning Disabilities, 5(4), 299-308.

Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, A. C. (1998). Violent offenders: Appraising and managing risk. Washington, DC:American Psychological Association.

Serper, M. R., Goldberg, B. R., Herman, K. G., Richarme, D., Chou, J., Dill, C. A., et al. (2005). Predictors of aggression on the psychiatric inpatient service. Comprehensive Psychiatry, 46, 121-127.

Simpson, M. & Hogg, J. (2001). Patterns of offending among people with intellectual disability: a systematic review. Part I: methodology and prevalence data. Journal of Intellectual Disability Research, 45(5), 384-396.

Vanderhoff, H. & Lynn, S. (2001). The assessment of self-mutilation: Issues and considerations. Journal of Threat Assessment, 1, 99-109.

Webster, C. D., Martin, M. L., Brink, J., Nicholls, T. L., & Middleton, C. (2004). Manual for the Short Term Assessment of Risk and Treatability (START). Version 1.0, Consultation edition. St. Joseph’s Healthcare, Hamilton, Ontario, Canada, and Forensic Psychiatric Services Commission, Port Coquitlam, British Columbia, Canada

Webster, C., Muller-Isberner, R. & Fransson, G. (2003). Violence risk assessment: using structured clinical guides professionally. International Journal of Forensic Mental Health, 1, 43-51.