Manual Evaluation:Are we doing what we think we are doing?

Associate Professor Dee ManginDirector Primary Care Research Unit

Department of Public Health and General PracticeChristchruch School of Medicine,

University of Otago

20 May 2009WHA

Overview

Rationale for testing the manual in a pilot study

Protocol for testing

Timelines

Background

WHO/HAI have developed a comprehensive teaching package

comprising a manual and accompanying resource material as well as

training for teachers implementing and this

Background

It is planned to evaluate this in a pilot study to

look at its effectiveness and at areas for

improvement in the final manual

WHY DO THIS STUDY?

Background

Many educational resources and techniques

are implemented without ever having any

assessment of their effectiveness

So we don’t know whether our efforts are effective

Overview

Our wish to think that they are can be misleading……

Many strategies for medical education have little effect on prescribing when they are tested in studies

Evidence for prescribing education strategies that have been found effective

The effect of dissemination only strategies such as didactic lectures and written information are small at best and of unknown clinical significanceFarmer et al. Cochrane Database of Systematic Reviews 2008;3

Facilitated teaching with evidence based resources and audit and feedback have been found to have a moderate effect Richards et al Family Practice 2000

A previous WHO educational resource (WHO Guide to Good Prescribing) using facilitated teaching for medical students on pharmacotherapy showed a significant effect on students skills in a randomised controlled trial Impact of a short course in pharmacotherapy for undergraduate medical students: An international randomised controlled study. de Vries T Henning R Hogerzeil H Bapna J et al The Lancet 1995

Study Design

• Randomised controlled trial

10 volunteer sites

5 intervention sites 5 control sites

• Quantitative and qualitative methods to assess change in students knowledge attitudes and skills

Questions

• What is the effect on students– knowledge, attitudes and skills of students in

understanding and responding to pharmaceutical promotion

• What are the strengths and weaknesses of the content of the manual and resources provided– assessed across a variety of settings

Data collection

•Before and after assessment of knowledge, attitudes and skills of students

• Questionnaire• Focus groups

•Key informant interviews and content evaluation by teachers in the intervention group

Two comparisons between these randomised intervention and control groups

The difference between the before and after manual assessments at the beginning and end of the year compared to the control sites

The difference between their assessments at the end of the year and students at the end of the year in the previous year groupAgain compared to the differences at the control sites

Comparison 1

Change in scores at intervention sites compared to control sites

5 5

intervention sites control sites

manual teaching as usual

Before Before

compared to

After After

difference difference

Comparison 2

• Difference in scores between study groups at the end of the year and students at the end of the previous year 5 5 intervention sites control sites

Students from previous year tested for comparison

Students from previous year tested for comparison

Students at end of study year

Students at end of study year

Study start Study start

Manual implemented Teaching as usual

Volunteer sites

• Medical or pharmacy students• Site prepared to teach all manual chapters• Site able to teach within one academic year (12

months)• Ethics approval will be gained where required

Volunteer sites

• Training, travel and accommodation will be provided for a teacher from each site

• The evaluation assessments and analysis will be organised and co-ordinated by the study team based at HAI

• These involve the before and after questionnaires and focus groups for students/interviews for teachers at each site

Volunteer sites: Timeline

• Plan to implement as academic years begin from later in 2009 through to 2010

• Training for teachers from sites randomised to intervention in Amsterdam Sept 2009

• ‘Before’ assessments as academic years start• ‘After’ assessments completed late 2010 as years

end• Training for teachers from sites randomised to

control in Amsterdam in Sept 2010

It is not new

We need to ensure medical and pharmacy graduates are well equipped to understand and deal with the effects of promotion on their prescribing decisions

It is important that we not just have good ideas

But that we know whether our collective efforts are effective

Volunteers!- Pilot study- Testing the manual in your setting- Expression of interest forms at the back

of the room OR- Email: lisa@haiweb.org