Evaluating complex interventions

(or the tribulations of trials)

Prof. Rowan Harwood

Health Care of Older People Nottingham University Hospitals

What we may be witnessing is not just ... the passing of a particular period of post-war history, but the end of history as such...

That is, the end point of mankind's ideological evolution and the universalization of Western liberal democracy as the final form of human government.

The end of Medical HistoryJohn R Hampton

Examples of complex services for older people

• Rehabilitation

(stroke, community hospitals, intermediate care)

• Most psychiatry

• Falls services

• Continence services

• Care home and long term care interventions

• Education, training, guidelines, care pathways, service improvement

intervention … comprising a number of separate elements which seem essential to its proper functioning;

components … may act independently and inter-dependently

usually include: • behaviours • parameters of behaviours (frequency, timing) • methods of organising and delivering those behaviours (types of practitioner, setting and location).

Complex interventions

MRC 2000

• Briefly describe 4 trials

• Highlight design, implementation, analysis and interpretation issues

• Consider design compromises in a new trial of a complex intervention constrained by service demands

Plan

Stroke Unit Evaluation 1991-3

• Operational since 1982, 15 beds

• Restrictive inclusion criteria• 315/1760 (18%) stroke patients included

• SU median 76 days, standard care 54 days

• Follow up: 275 (87%) at 3 mo; 241 (77%) at 12 mo

• At 12mo: • Died 14% vs 19%

• Home 65% vs 62%

• Barthel 17 vs 16

• Rivermead ADL 12 vs 9

• NEADL 7 vs 4

• GHQ 17 vs 20

• psychological adjustment better at 3 & 6 months

Stroke Unit Evaluation issues

• Keep unit full (changed to 60:40 randomisation)

• Blinded assessment

• Loss to follow up (23% at 12 months)

• Contamination

• Power

• Multiple outcome measures

• … but what is a stroke unit?

• … why the difference in outcome?

SUE observation of process

• Non participant observations of 76 patients

• 3x8h shifts over 3 days, 10min intervals

• location• rehab 6.4% vs 3%; bedside 61% vs 65%; social 5% vs 2%

• activity• isolated disengagement 15% vs 24%

• positioning - poor 13% vs 30%

• contacts• more time with nurses, therapists; less time with doctors, visitors

• rehab time (on ward) 46 min/d vs 23min/d

• Established multidisciplinary service

• 428 referrals randomised (50:50 then 40:60) over 2y

• Group A: routine care including day hospital or OP (n=189)

• Group B: CST (n= 232). Median 18 sessions (range 0-115, IQR 10-42). Mean 4.8 h PT; 3.8h OT; 2h SLT, 1.9h MH nurse; 0.5h Rehab Support

• Six months later, contacted by letter, requested consent to follow up, postal outcome questionnaire

• Consent procedure:• avoid perception of ‘missing out’ by control group

• reduce team loyalty bias

• logistically easier

• Offer of assistance or visit to complete questionnaire

Community Stroke team

Community Stroke Service trial: issues

PatientCST vs standard

Barthel/20 16 vs 16NEADL/44 24 vs 26GHQ/24 13 vs 15EQ5D/100 52 vs 55Satisfaction

emotional support/4 3 vs 2 (CST better)overall/4 3 vs 2 (CST better)

Carer

GHQ/24 13 vs 15Carer strain/12 8 vs 10 (CST better)EQ5D 73 vs 75satisfaction

knowledge/4 3 vs 2 (CST better) overall/4 3 vs 2 (CST better)

Outcomes at 6 months

• Low and differential follow up (47% vs 43%)

• No baseline data

• No record of routine services

• Time but not content of CST intervention recorded

• Was the service successful or not?

• If this didn’t work what could …?

Community Stroke Team: problems

A new EDRS was established in 1998.

9-month development phase.

Staffing: 2 OTs, 2 PTs, 3 nurses, 1 SW, 7 rehab assistants, secretarial support. No doctors. Visits for assessment, monitoring, rehabilitation, or assistance and care.

Up to four visits per day, up to 7 days per week, 8am to 10pm.

Duration up to 4 weeks.

EDRS: service structure

• 370 elderly medical, stroke and surgical patients

• Randomised between EDRS and conventional care

• Mean 22 visits per patient over 4 week

• Mean 7 days shorter hospital length of stay

EDRS: randomised trial

EDRS: outcomes

Qualitative evaluations

• Exploratory, in depth

• Subject to ‘rules’

• Identify:• what an intervention comprises in practice • how an intervention is useful• how it achieves its effect• the conditions necessary for its effect

• Introduce rigour into otherwise potentially anecdotal evidence

• Patients interviewed:• before hospital discharge

• after 4 weeks

• after 3 months.

• Staff responsible interviewed 6–8 weeks later

• 64 interviews of 20 patients

• 15 interviews of 11 staff

EDRS: Qualitative study

• Before discharge• Participants anxious about return home, but keen to do so

• Felt vulnerable, feared not being able to cope, or being a burden on families.

• After discharge• Patients felt they had improved

• … but described pain, loneliness, fatigue, reduced mobility, problems climbing stairs or getting outdoors.

• Lack of confidence and anxiety about activities

Qualitative study

• EDRS delivered: skilled assessment, negotiated treatment goals meaningful to the patient, and demonstrated team working.

• Interventions: functional rehabilitation, teaching skills, information giving, advice, overcoming emotional barriers, provision of aids and appliances and provision of personal and domestic care.

• Emotional support: improved confidence and morale, and led to greater task performance and psychological well-being.

Qualitative study

• 1 intervention ward, 2 standard care wards

• Multi-component delirium prevention regimen

• hydration

• vision and hearing

• orientation

• mobilisation

• non-pharmacological sleep promotion.

• RCT pilot failed - ‘120% bed occupancy’

• Prospective individual matching design

Delirium prevention trial

• Prospectively assessed: 2424 patients admitted to 3 wards

• Research staff ‘not involved in care’, ‘unaware of nature of study, or patient group assignment’.

• Over 70, at intermediate or high risk of delirium

• Recruited all patients on intervention ward

• Matched these with standard care patients• 5 years

• sex

• baseline risk of delirium

• U&E, visual and cognitive impairment, APACHE-II.

Prospective individual matching design

• Assessed 2424 patients

• Excluded 1265• severe dementia 154

• aphasia 38

• no English 92

• coma or terminal illness 69

• LOS<48h 219

• other incl. no interviewer or patient unavailable 355

• Declined 250

• No match 67 (24 intervention, 43 standard)

• Recruited 852 (426 intervention, 426 standard)

Exclusions

Results

• Able to do research interviews on all at admission

• Demonstrated power equivalence of RCT

• Recruitment

• Demonstrated good matching at baseline on 32 variables

• Contamination

• Bias and blinding

Issues

MRC 2000 framework

MRC 2008

NIHR Medical and mental health unitcontrolled clinical trial

Medical and mental health unit: development

• Initial 250 patient register/cohort study.

• Multi-professional operational group

• Additional staff: RMNs and MH specialist OT/PT

• Programme of staff training

• Advice from MH service

• Environmental changes

• New approach to families

• Develop policies and new working practices

Medical and mental health unit

‘sufficiently different from standard care to have a reasonable chance of demonstrating different outcomes in a trial’

Medical and mental health unit: controlled clinical trial

• Modelled on stroke unit trials

• Compared with ‘standard care’

• Starting May 2010 (pilot); July 2010 (main trial)

• up to 2 year recruiting, 3 months follow up

• Target n=240 per group

• Effectiveness, economic and qualitative studies:

• not just does it work, but how does it work?

• is it value for money?

• Service funding ceases April 2013

Medical and mental health unit: controlled trial

Outcomes at 3 months:

• number of days spent at home:• length of stay, readmissions, deaths, care home placements

• number of days in care home for those starting there

• scales:• DemQoL, NPI (behaviour), Barthel, EQ5D, London Handicap

• Carer strain and psychological wellbeing

• resource use and costs

• non-participant observer study

Medical and mental health unit: recruitment and randomisation

Constraints

• No waiting on AMU (for assessment/recruitment/consent/MMHU bed)

• No empty beds on MMHU

• Keep MMHU full with suitable patients

• Minimise ward moves for confused patients

• Likely to lose trial beds to non-randomised patients

Medical and mental health unit: Recruitment and randomisation

Randomisation prior to consent, recruit to follow up later.

• ‘Confused, over 65’

• Web based stratification (care home or not)

• Transferred to MMHU or standard ward

• Once on ward recruited to trial (consecutively or randomly)

Other issues

• Need to persuade AMU to do something of no benefit to them

• Majority lack capacity to consent: effort and delay

• 10% lack a consultee

• 40% of cohort declined

CLRN/MHRN to the rescue

• Initial plan was limited by availability of researchers to recruit and follow up

• Current plan to recruit all randomised patients who agree to take part

• Option to collect primary outcome on those declining full study

Summary and lessons for the triallist

• The RCT can be adapted to evaluate complex interventions

• Logistics can be tricky and compromises needed

• Interpretation is more difficult

• multiple outcome measures

• what is the outcome?

• does the outcome match the intervention?

• condition specific vs generic outcomes

• qualitative evaluation

• Look at the content of the intervention

• what is the intervention?

• how different it is from the comparator

Complex interventions

Thank you.

Watch this space!