Presented by Jennifer Holmes Drug & Alcohol Service

Langton Centre

May 2012

The Australian Treatment Outcome Profile (ATOP): Clinical Monitoring for the Drug & Alcohol

SectorAnni Ryan, Adrian Dunlop, Jennifer Holmes, Vi Hunt, Kristie Mammen, Raka Tierney, Nicholas Lintzeris.

Acknowledgements

Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty Jago, Skye O’Donnell, Gonzalo Rivas, Terry Schofield and Doungkamol Sindhusake.

MHDAO, Ministry of Health, D&A Research Program Grant 2009/2010

MHDAO, Ministry of Health, Research Grant 2011/2012

UK National Treatment Agency, NHS

Overview

Background to the ATOP

Stage 1: Validating the ATOP in Australian populations

Stage 2: Expanding across services in NSW

Stage 3a: Australian dissemination / uptake

?Stage 3b: Developing Composite Measure of Treatment Outcome (ATOP Treatment Episode Global Outcome)

A clinician administered 1-page validated ‘instrument’

Self-report measures across 2 key domains in the preceding 4 weeks

– Substance use

– Health & well being

– Global ratings (0-10) of physical & mental health, quality of life

– Housing, employment & study, violence, child protection

What is the ATOP?

Benefits of routine clinical outcome monitoring in health services are well documented – clinician administered e.g. HoNOS– repeat blood tests (CD4 counts in HIV)

Few examples of successful implementation of in D&A services

Background: Clinical Outcome Monitoring

length of the instruments (e.g. >6−10 pages long, taking >20 minutes to complete)

+

inadequate attention to training

+

excessive data entry requirements

+

poor feedback for patients, clinicians and administrators

=

limited utility and resistance

Why is implementation so difficult?

Treatment Outcome Profile (TOP) – a validated tool introduced by the National Treatment

Agency in the UK in 2007– introduced across all NHS funded D&A services in

England

Modified for Australian conditions = ATOP– Reflecting substances commonly used in Australia – Made more ‘intuitive’ to complete – Validated in Australian populations & against common

Australian instruments

Developing the ATOP

to assist in systematic, documented and ongoing client assessment

to provide feedback to clients about ‘progress’ over time, and assist in treatment care planning and motivation

to assist in clinical handover when transferring/referring clients to other services

to assist in service evaluation – helping us to answer the broader questions of “do our clients get better?”

as a quality and potential research tool with services collecting standardised data

Why use the ATOP? Different purposes for different people

Stage One: Project Aims

Validate the Treatment Outcome Profile (TOP) under Australian conditions

Examine implementation and feasibility issues in 3 NSW public OPT clinics

What do we mean by ‘validated scale’

Face validity: “does it look right”

Does it measure things as well as other instruments

– K-10, WHO-QOL, SF-12, PHQ-15

– OTI (substance use, injecting practices)

Inter-rater reliability:

– do 2 different raters administering the instrument on the same client get the same score?

Methods

Concurrent validity and inter-rater reliability:

ATOP administered as part of routine care by clinic staff at 3 month intervals

Research interview within 72 hrs of last ATOP: gold standard instruments + repeat ATOP

Implementation and feasibility issues (incl. data management):

clinician, service manager and patient perspectives

satisfaction surveys and focus groups

Results - client demographics (n=104)

GenderMaleFemale

62 (59.6%)42 (40.4%)

Age 0-3031-4041-5051+

32 (30.8%)37 (35.6%)22 (21.2%)13 (12.5%)

Results – concurrent validity

Concurrent validity Kappa Spearman’s

Section 2: Injecting Risk Behaviour    

Injected with a needle/syringe used by someone else

0.81 (p<0.001)

0.83 9p<0.001)

Section 3: Crime    

Drug Selling 1 (p<0.001) 1 (p<0.001)

Committing Assault or violence 1 (p<0.001) 1 (p<.0001)

Results - concurrent validity

Section 4: Health and Social Functioning Spearman’s

Psychological Health  

vs WHOQuOL Q26 0.62 (p<0.001)

vs K10 total -0.60 (p<0.001)

 

Physical Health  

vs PHQ15 Total -0.58 (p<0.001)

SF12 Physical Component Score total 0.65 (p<0.001)

   

Quality of Life  

vs WHOQuOL Q1 0.69 (p<0.001)

vs WHOQuOL Social Relationship Domain total 0.55 (p<0.001)

vs WHOQuOL Environment Domain Total 0.66 (p<0.001)

vs SF12 Physical Component Score total 0.35 (p<0.001)

vs SF12 Mental Component Score total 0.62 (p<0.001)

Results - inter-rater reliability (n=103)

No significant difference between researcher mean and clinician mean on all continuous items

All dichotomous items correlated (p<0.001)

Clinician feedback (n=20)

easy to administer: 80% agreed

appropriate for my client population: 85% agreed

format & style easy to understand: 85% agreed

length appropriate for routine practice: 70% agreed

appropriate for my setting: 65% agreed, 30% unsure

useful in developing a case plan: 65% agreed, 25% unsure

useful for identifying important problems 45% agreed, 35% ambivalent, 20% disagree

happy to use as part of regular client reviews 45% agreed, 30% ambivalent, 25% disagree

Dislike… crime questions

“too many, too difficult, all clients report nil…”

“Not sure about the crime questions, although they are important.”

At first ATOP clients say “no” to crime questions but by 3 to 6 months clients start to answer “yes”

Clients express concerns over privacy/confidentiality/how this information may be used esp. where there is legal/child protection issues

Client feedback (n=123)

ATOP questions were easy to understand 93% agreed

Helpful way of looking at how well treatment is working for me 85% agreed

Helpful to have this same review every few months 85% agreed

Length of the ATOP was “about right” 90% agreed

Other comments…

“ I don’t think everyone will be honest about the crime questions…”

“asks the right questions: interesting to do it again…”

“some questions feel like they are a stereotype of drug users…”

“I think there should be more questions about how much the client feels the program is helping them…”

Conclusions Stage One

ATOP is a valid instrument for measuring treatment outcomes in an Australian opioid maintenance treatment population

ATOP is compatible with routine clinical practice

ATOP can feasibly be implemented as part of routine clinical practice in public OPT Clinics

Investigate feasibility of ATOP in other treatment settings

Stage 2 Pilot across NSW Services

Beyond OTP

Training

Protocol & Business Rules

Data Systems

Stage 2 Currently trialled across 10-15 sites

– Counselling

– OTP

– Withdrawal Management

Feasilbility and clinician feedback

Establishing protocol and business rules for services

Training Package

– Train the trainer workshops

– Presentations and training manual

Putting the ATOP into Practice: a crash course for clinicians and

service managers

How to complete the ATOP

Enter– Client MRN, date of birth and sex– Your name– Date of ATOP– The stage at which the ATOP is being done

Enter client responses– Use the Conversion Table to calculate standard drinks – Nil drug/alcohol use – enter “00” in the total box– Timeline – invite the client to recall the number of days in each of the past four weeks on

which they did something – Before asking Section 2: Items (e) to (h) remind the client about confidentiality (see box)– Yes and no – a simple tick for yes or no– Rating scale – a 10-point scale from poor to good. Together with the client, CIRCLE a

number. – Refused/can’t recall – write “NA” (short for Not Answered) next to the total box, tick box or

rating scale.

ATOP across Australia ATOP being explored/used in

– South Australia (DASSA)

– Tasmania (Tas govt sector)

– WA

– Victoria (Turning Point)

– NADA data set (NSW NGO sector)

Designing a Composite Measure of clinical outcomes in

D&A treatment The ATOP Research Team

March 2012

Composite performance measures

The combination of 2 or more indicators into a single number to summarize multiple dimensions of performance and to facilitate comparisons.

– integrate a large amount of information in a format that is easily understood.

– increasing use by governments to assess performance.

– examples: Dow Jones Index, IQ ratings, NAPLAN, SF-36

In substance abuse treatment

Challenge: To establish a measure that summarises multiple clinical outcomes commonly associated with D&A treatment

– Substance use (primary & secondary drugs)

– General health & well-being (physical health, mental health, overall QOL)

– High-risk behaviours: injecting, homelessness, child protection, violence, crime

Why a composite outcome index

To be able to broadly describe whether or not a treatment episode is associated with changes in patient well-being

To be able to state for a particular program:

– 56% patients significantly improved

– 27% had no significant change

– 17% deteriorated

Allows identification of benchmarks by:

– Drug type (alcohol, opiates etc…)

– Treatment type (e.g. counselling, OTP, withdrawal etc…)

– Patient factors (e.g. ATSI, age, rurality)

Examples to date: opioid treatment

Dutch Heroin Trial (van den Brink et al. 2003 BMJ 327)

Used a pre-specified dichotomous, multidomain outcome index as the primary outcome parameter. Patients considered ‘responders’ if they showed – at least 40% improvement in at least one of the three

domains (physical, mental, social) at end of treatment compared with baseline;

– if this improvement was not at the expense of a serious ( ≥ 40%) deterioration in functioning in any of the other outcome domains; and

– if the improvement was not accompanied by a substantial ( ≥ 20%) increase in use of cocaine or amphetamines.

Examples to date: cocaine treatment

Paul Crits-Christoph et al. Psychosocial Treatments for Cocaine Dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Arch Gen Psychiatry. 1999;56:493-502

Composite Cocaine Use Measure

– pool information from multiple measures (urine drug tests, ASIs, and weekly cocaine use inventory) to code each month of treatment as abstinent vs not abstinent. Any indication of cocaine use from the 3 measures would lead to a “not abstinent” month.

Composite Psychiatric Measure

– To test hypothesis: degree of psychiatric symptoms interacting with the treatment condition, a composite measure of 4 scales—the Hamilton Rating Scale for Depression, the Beck Anxiety Inventory, the Brief Symptom Inventory, and the ASI–Psychiatric Severity Composite score—was created by converting each scale to a standard score and then averaging the scores.

Examples to date: Alcohol treatment Zweben & Cisler. 2003. Clinical and Methodological Utility of a Composite Outcome Measure for Alcohol Treatment Research. Alcoholism: Clinical and Experimental Research 27 (10), 1680–1685

Project MATCH data: composite outcome measure to capture multiple treatment outcomes among diverse client populations.

– self-reported alcohol consumption, alcohol problems, biological markers, other areas of functioning (psychiatric dysfunction, QOL).

Findings on composite measure:

– 30% clients sustained a ‘remitted’ status (i.e., abstinent or moderate drinking without problems) over 1-year follow-up;

– 70% of the clients had reached a nonremitted status (i.e., heavy drinking and/or problems).

“The composite outcome index could be used usefully along with singular measures of consumption to obtain a more complete picture of what has occurred among clients”.

Methods for combining items into a Composite Measure

Linear combinations – Composite = [indicator1×weight1] + [indicator2×weight2] + . . . + [indicatorN×weightN] – Expert Panel determine which indicators & which weightings

Regression-Based Composite Measures – If a certain outcome is a ‘gold standard’, the weighting of individual items may be

determined empirically by optimizing the predictability of the gold standard end point. Latent Trait Composite Measures

– Identify clusters of correlated items & latent trait modeling may be used to combine items within clusters but not across clusters.

– e.g. substance use outcomes; general health/well-being; high risk / harm behaviours Any-or-none Scoring of outcome measures

– In this method, a patient is counted as failing if he or she experiences at least 1 adverse outcome from a list of 2 or more adverse outcomes.

Opportunity Scoring – Opportunity scoring counts the number of times a given care process was actually

performed (numerator), divided by the number of chances a provider had to give this care correctly (denominator).

– e.g. individual treatment care plan completed; MH-COPES completed All-or-None Scoring of Process Measures

– Only those patients who receive all indicated processes of care are counted as successes.

Recommendations re: Developing Composite Measures (Peterson et al 2010)

1. The intended audience & purpose of a composite measure should be explicitly stated.

2. Decisions about which measures should be based on clinical importance of patient outcomes and the reliability of individual performance measures.

3. Each individual component should be precisely defined to ensure consistent application in different settings.

4. The description of the methods used for weighting and combining individual measures into a composite performance measure should be transparent.

5. Developers should explore a variety of alternative methods for combining measures and document whether conclusions about provider performance differ with use of alternative methods.

6. Empirical testing needed to assess the properties of a composite measure score.

7. Reporting of composite performance measures should be accompanied by detailed reporting of individual domains and components.

8. Reporting of composite performance measures should include a measure of the degree of uncertainty surrounding composite estimates for providers.

9. Composite performance measures must be reevaluated as that evidence changes.

Could the ATOP be used to develop a composite measure of D&A treatment outcome / success?

Instrument needs to be used by the sector

Instrument shown to be robust across diverse populations, drug types, treatment types

Measures the main domains that we are most interested

– in over time (e.g. beginning & end of treatment)

– substance use, general health, high-risk behaviours

Linked to other aspects of NMDS

– identifying primary/secondary drug use, demographics, treatment types delivered )

ATOP Treatment Episode Global Outcome

Aim

To develop & validate the ‘ATOP Treatment Episode Global Outcome’, as a means of assigning a global outcome for each D&A treatment episode that broadly reflects whether each treatment episode was associated with a significant improvement, no change, or significant deterioration in the main clinical domains.

Proposed methodologyATOP Treatment Episode Global Outcome

National, multisite, project, 2-3 year duration – Reference group of expert clinicians, consumer reps, researchers, data experts

Recruit large number of clients (100’s) entering a variety of treatment types (counselling, OTP, withdrawal, rehab) and using different primary drug types (alcohol, opioids, cannabis, other) – ATOP at beginning & end of treatment episode (or intervals for OTP, rehab), &– “Gold standard”: client & clinician global outcome ratings for each treatment

episode (confirmed by independent clinician & client panel, with broad criteria identified by Reference group for different treatment types).

Identify algorithms for attributing ATOP Treatment Episode Global Outcome for different treatment modalities & different primary drug types. Statistical methods (e.g. RUC) to identify algorithms against ‘gold standard’ measure of client & clinician global ratings. – How much change in (a) substance use, (b) general health & (c) high-risk

practices is required for the ATOP scores to match client & clinician global ratings.

The holy grail?

Data systems that enable us to triangulate data re: – client characteristics

– services provided (treatment episode data such as type of service, number of contacts, by whom)

– client outcome

Would enable benchmarking of services

NMDS possibly only mechanism to ensure this occurs.

For Further Information Contact ATOP Project Coordinator

– Kristie Mammen– Kristie.Mammen@SESIAHS.health.nsw.gov.au– Langton Centre 02 93328777

Composite Measure Project– Nicholas Lintzeris

Validation of ATOP– Anni Ryan

Data Collection Systems– Jennifer Holmes– Jennifer.Holmes@SydneyMSIC.com– 02 9360 1191