DEPARTMENT OF HEALTH AND AGEING …...Health Outcomes International The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage

Health Outcomes International Suites 1 & 4, 51 Stephen Terrace, Kent Town SA 5069

Phone: 08 8363 3699

Facsimile: 08 8363 9011

Email: [email protected]

ABN 80 081 950 692

DEPARTMENT OF HEALTH AND

AGEING

EVALUATION OF PROGRAM RESOURCES, TRAINING MATERIALS, INCLUDING DELIVERY

FORMATS, DEVELOPED FOR THE

IMPLEMENTATION OF STAGE 2 OF THE

DIABETES MEDICATION ASSISTANCE SERVICE

EVALUATION REPORT

JUNE 2010

mailto:[email protected]

Health Outcomes International

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 ii

C CONTENTS

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

E.1.1 Introduction .................................................................................................................................................. 1

E.1.2 Evaluation Methodology ............................................................................................................................ 1

E.1.3 Key Findings ................................................................................................................................................ 1

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1 The Diabetes Medication Assistance Service .......................................................................................... 3

1.2 The Training and Credentialing Process .................................................................................................. 3

1.3 Evaluation Objective ................................................................................................................................... 4

1.4 The Structure of the Proposal ................................................................................................................... 4

EVALUATION METHODOLOG Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Documentation Review and Analysis ....................................................................................................... 5

2.2 Technical Analysis ....................................................................................................................................... 5

2.3 Survey of Community Pharmacist ............................................................................................................ 5

2.4 Stakeholder Consultation ........................................................................................................................... 6

2.4.1 Program Partners ......................................................................................................................................... 6

2.4.2 Community Pharmacists............................................................................................................................. 7

PROGRAM RESOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.1 Program Resources ...................................................................................................................................... 8

3.2 Evaluation findings ...................................................................................................................................... 9

3.3 Opportunities for improvement.............................................................................................................. 11

3.3.1 Ease of use .................................................................................................................................................. 11

3.3.2 Content – lifestyle & technical ................................................................................................................ 12

3.3.3 Administration & Other ........................................................................................................................... 12

TRAINING DELIVERY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.1 Evaluation findings .................................................................................................................................... 14


TRAINING MATERIALS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19



The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 iii


CREDENTIALING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22



CLINICAL APPROPRIATEN ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

7.1 Overall conclusions from a clinical perspective.................................................................................... 24


7.2.1 Qualifications of the program resources development team.............................................................. 25

7.2.2 Qualifications of the training materials development team ................................................................ 26

7.2.3 The training content .................................................................................................................................. 26

7.2.4 The delivery format ................................................................................................................................... 27

7.2.5 The credentialing process ......................................................................................................................... 29

7.2.6 Gaps and opportunities for improvement ............................................................................................. 30

OTHER FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

8.1 Cost effectiveness ...................................................................................................................................... 32

8.2 Withdrawing pharmacists ......................................................................................................................... 32

8.3 Post training & implementation support/Mentors .............................................................................. 32

8.4 Training subsidy ......................................................................................................................................... 32

APPENDIX A – QUANTITATIVE SURVEY RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I


The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 1

E EXECUTIVE SUMMARY

This report summarises the work completed and our findings in relation to the evaluation of the effectiveness and appropriateness of the: program resources, format for training delivery, associated training materials and training credentialing process of the DMAS training program.

E.1.1 INTRODUCTION

The Diabetes Medication Assistance Service (DMAS) is a professional pharmacy service designed to assist patients with poorly controlled type 2 diabetes to improve self-management of their condition through a series of in-pharmacy consultations with a pharmacist.

To participate in the DMAS, it is mandatory for pharmacists to complete a process of training and credentialing. Training involved Pharmacists completing a series of four pre-workshop reading modules and attending a two day face-to-face training session; and undertaking a series of online multiple choice questions. Credentialing for pharmacists involved interpretation of blood glucose readings and completion of a series of short answer questions whilst at the two day training session. This is followed by the Pharmacist completing online a series of multiple choice questions within seven days of the training. Credentialing was assessed by the AACP.

E.1.2 EVALUATION METHODOLOGY

The Evaluation Framework (April 2010) set out the key components of the work to be performed. In summary our approach included:

Consulting with key stakeholders in relation to the development, design and delivery of the training program.

Consulting with 20 pharmacists participating in the DMAS program to ascertain their views on the training program.

Conducting an online survey of all credentialed pharmacists (achieving 176 responses, 22% of credentialed pharmacists) to ascertain their views on the training program.

Engaging a team of clinical experts to review the clinical appropriateness of the training content and materials

Consulting with 5 pharmacists who withdrew after the training and credentialing process

Reviewing relevant documentation and material produced for the training and credentialing process

E.1.3 KEY FINDINGS

It is our view that each component of the DMAS training program (program resources, format for training delivery, associated training materials and training credentialing process) was effective and appropriate.



Despite this overall positive conclusion, a number of areas for improvement have been identified through the relevant sections of this report. We have summarised the key improvements below:

The composition of the teams responsible for developing the training materials and program resources could have been strengthened by the addition of:

o A representative of GPs,

o A consumer representation, a

o A endocrinologist.

Pharmacists commented that they wanted more material in relation to diet and exercise. Our discussions with pharmacists identified that these matters were often key areas of focus and the inclusion of additional material for the pharmacists, but also materials for the patient would have been very useful.

Provide pharmacists with a single electronic source in relation to program resources and information (rather than multiple sources). It was suggested that a single electronic or online source reduced paper and could be readily updated. This would incorporate electronic forms that could be stored electronically. Similarly, pharmacists would have preferred to have a single website to access information on the program (rather than multiple).

Pharmacists would have appreciated more training in methods to overcome implementation barriers. In particular, the engagement of GPs was identified as a barrier by some pharmacists. Some suggestions to improve this in future programs were to include more literature in relation to patient/doctor management, to enable them to provide more information to GPs. Some post-training support from a “facilitator” between pharmacists and GPs may have assisted.

Face to face time with a diabetes educator (rather than the DVD) would have assisted the pharmacists‟ knowledge development. Pharmacists noted that most patients had seen a diabetes educator, and it would be good to have done the same. In addition, the inclusion of a diabetes educator in the assessment of BGM uses may have strengthened pharmacists competencies.

A focus on practical elements of implementing and running the program could have been achieved by including training elements such as:

o More case studies

o Role playing opportunities or patient scenario re-enactments. Effective patient consultation, one on one, is new to most pharmacists.

It would have been clearer and apt if the clinical information in the training materials was customised for the Australian setting, as opposed to inclusion of information related to overseas countries. This would better suit the needs of the community pharmacists delivering DMAS. In addition, we feel that Australian cardiovascular risk calculators/charts and their interpretation could be introduced to the pharmacists during the pre-workshop modules or workshops.

Inclusion of a mentor program to provide post-training and continued support to credentialed pharmacists may have allowed for a more effective implementation of the program into the workplace.

We note that we consulted with five pharmacists that withdrew from the DMAS program. The withdrawing pharmacists spoke favourably of the training and their views were generally consistent with the survey results presented in this report.



1

INTRODUCTION

The Australian Government Department of Health and Ageing (the Department) appointed Health Outcomes International (HOI) in March 2010 to undertake a comprehensive evaluation of the effectiveness and appropriateness of the program resources, training materials, including delivery formats, developed for the implementation of Stage 2 of the Diabetes Pilot Program. This paper presents the final evaluation report in relation to the evaluation.

1.1 THE DIABETES MEDICATION ASSISTANCE SERVICE

The Diabetes Medication Assistance Service (DMAS) is a professional pharmacy service designed to assist patients with poorly controlled type 2 diabetes to improve self-management of their condition through a series of in-pharmacy consultations with a specially trained and credentialed pharmacist. The pharmacist assesses patients over a 6 month period and assists the patient in diabetes self-management through providing help with blood glucose meter management, monitoring of blood glucose, supporting the Quality Use of Medicines with a focus on medication adherence, and measuring blood pressure and lifestyle issues.

The DMAS is the first program funded through the community pharmacy agreements that has had a significant training component. Through the conduct of this project, there is potential to gain insight to enhance the effectiveness of the DMAS training components and inform other future programs.

1.2 THE TRAINING AND CREDENTIALING PROCESS

To participate in the DMAS, it is mandatory for pharmacists to complete a process of training and credentialing.

Training was conducted by the Pharmaceutical Society of Australia (PSA). This involved Pharmacists completing a series of four pre-workshop reading modules and attending a two day face-to-face training session; and undertaking a series of online multiple choice questions. An overview of the content is as follows:

Preparatory work (pre-reading modules):

Module 1 – Diabetes basics and pathophysiology of tyre 2 diabetes

Module 2 – Complications of type 2 diabetes

Module 3 – Pharmacotherapy in type 2 diabetes

Module 4 – Lifestyle management for type 2 diabetes

Face to face training (2 day workshop) covering the topics:

DMAS protocol

Quality use of medicines

Consultation skills and counselling interventions

Goal setting for DMAS



Hands-on devices workshops

Practice tools, resources and documentation

Credentialing for pharmacists to participate in DMAS was undertaken by the Australian Association of Consultant Pharmacy (AACP). The credentialing involves interpretation of blood glucose readings and completion of a series of short answer questions whilst at the two day training session. This is followed by the Pharmacist completing online a series of multiple choice questions within seven days of the PSA training which are assessed by the AACP.

1.3 EVALUATION OBJECTIVE

The overall objective of the evaluation is to assess the effectiveness and appropriateness of the:

1. program resources

2. format for training delivery

3. associated training materials

4. training credentialing process

These objectives will be supported through exploration of key evaluation questions that will assess the overall appropriateness and effectiveness of the DMAS training and credentialing processes.

1.4 THE STRUCTURE OF THE PROPOSAL

This report presents our findings and conclusions in relation to the evaluation objectives. The structure of this proposal is as follows:

Chapter 2: Describes our methodology for conducting the evaluation, and the stakeholders and participant pharmacists consulted.

Chapter 3: Presents our findings in relation to the effectiveness and appropriateness of the program resources.

Chapter 4: Presents our findings in relation to the effectiveness and appropriateness of the format of the training delivery.

Chapter 5: Presents our findings in relation to the effectiveness and appropriateness of the associated training materials.

Chapter 6: Presents our findings in relation to the effectiveness and appropriateness of the training credentialing process.

Chapter 7: Presents the findings of the Clinical Expert in relation to the clinical appropriateness of the program resources, training materials, including delivery formats

Chapter 8: Presents a summary of other findings and associated observations in relation to the effectiveness and appropriateness of the DMAS training program.



2

EVALUATION METHODOLOGY

A detailed evaluation methodology was developed for this project and is documented in the Evaluation Framework. This chapter summarises the key components of that framework.

2.1 DOCUMENTATION REVIEW AND ANALYSIS

We collected and reviewed appropriate program documentation to assist us in answering the relevant evaluation questions relating to the appropriateness and effectiveness of the training and credentialing process.

The documentation collected and reviewed comprised:

Training materials, including speaker notes

Pharmacy Readiness Kit

Online clinical material at the PSA and PGA websites

Accreditation materials

2.2 TECHNICAL ANALYSIS

HOI engaged a „Clinical Expert‟ to review all of the program resources, training materials and the credentialing process in respect to their clinical appropriateness. The Clinical Expert comprised a team of appropriately qualified personnel currently working at the Queen Elizabeth Hospital in South Australia. The team members comprised:

Personnel Position

Mahipal Sinnollareddy Clinical Pharmacist Registrar, The Queen Elizabeth Hospital

Cher-Rin Chong Clinical Pharmacist Registrar, The Queen Elizabeth Hospital

Hayley Vasileff Clinical Pharmacist Registrar, The Queen Elizabeth Hospital

Dr Irfan Aziz Endocrinology Registrar, The Queen Elizabeth Hospital

2.3 SURVEY OF COMMUNITY PHARMACIST

HOI conducted a survey of all 785 community pharmacists trained and credentialed. The survey was piloted on a small sample of pharmacists, before being released online (via Surveygizmo) on 19 May 2010. The survey was open for a period of two weeks, closing on 2 June 2010.

The results of the online survey comprise the views of 176 pharmacists (representing 22.17% of the 785 credentialed pharmacists). This number includes the 20 pharmacists subject to telephone consultation (as discussed below), as their views were entered into the survey by the HOI project team.



The locations (by state) of pharmacists participating in the survey are identified in figure 2.1 below. The proportion of respondents to the survey were in most cases relatively well aligned to the proportion of pharmacists credentialed in each state: ACT (1.78%), WA (9.3%) VIC (21.91%), TAS (1.91%), SA (13.38%) QLD (21.15%), NSW (29.68%) and NT (0.89%).

WA, SA, and NT were under-represented whilst Tasmania and the ACT were over-represented.

Figure 2.1: Percentage of survey respondents by state

A high proportion of survey respondents (83%) had been successful in recruiting at least one patient to the program and therefore had the opportunity to put the training into action. This places them in an appropriate position to provide feedback in respect to the appropriateness and effectiveness of the training in actually delivering the DMAS program.

Table 2.1: Percentage of respondents by number of patients

Number of patients % of respondents

No patients recruited 17.0%

1 patient 15.9%

2 to 5 patients 44.9%

More than 5 patients 22.2%

2.4 STAKEHOLDER CONSULTATION

In addition to the survey of community pharmacists, consultations were held with two key stakeholder groups; the program partners and the community pharmacists.

2.4.1 PROGRAM PARTNERS

Stakeholder telephone consultations were held with key personnel from each of the major partners involved in the process of developing and implementing the training and credentialing process for the community pharmacists participating in DMAS.



The personnel consulted were:

Stakeholder Group Personnel Consulted

Pharmacy Guild Australia (PGA) Erica Vowles & Jessica Rynehart

Pharmaceutical Society of Australia (PSA) Audra Millis

Australian Association of Consultant Pharmacy (AACP) Sarah Gillespie

Department of Health and Ageing Jacquie Maycock, Karen Farquhar, Joanne Oakman

2.4.2 COMMUNITY PHARMACISTS

Telephone consults were conducted with a sample of pharmacists (20) representing each State/Territory who have been successful in recruiting patients to the program. It was considered important to select from each State/Territory on the basis that the same trainers were not utilised in each jurisdiction and hence any potential differences that may exist could be contrasted.

Some participants selected in the initial sample were unable to be located or unwilling to participate in the telephone consultation. These were replaced in the sample of pharmacists by randomly selecting another pharmacist from the same sample pool.

In addition, consultations were held with a small sample (5) of those pharmacists who withdrew from the program following completion of the training. The purpose of this was to ascertain whether any pharmacist withdrew as a result of feeling inadequately trained to provide the program despite having undergone the theoretical and practical training.



3

PROGRAM RESOURCES

This chapter identifies our findings and areas for improvement in relation to the program resources developed and utilised in the DMAS program.

3.1 PROGRAM RESOURCES

“Program Resources” refers to the materials provided to, or made available to, pharmacists with respect to their participation in the training and ongoing service provision - in relation to Stage 2 of the Diabetes Medication Assistance Service. The Program Resources include:

Pharmacy Readiness Kit comprising:

Operations Manual

Pharmacy Forms Toolkit

Professional Guidelines and Standards

Promotional Materials

Evaluation Information

Clinical support materials provided on CD, and also available online in the Clinical Resource Centre (http://www.psa.org.au/site.php?id=3122)

Further program related information available on the Pharmacy Guild of Australia‟s website (http://www.guild.org.au/pps/content.asp?id=1896)

Table 3.1 sets out the key questions presented in the Evaluation Framework in relation to the evaluation of the program resources.

Table 3.1: Program resources evaluation questions

Evaluation Focus

Key Questions Domain

Program Resources

Was an appropriate mechanism utilised and appropriate partners engaged in the development of the program resources?

Appropriateness

In terms of appropriateness, how do DMAS program resources compare to other Fourth Agreement professional programs and services?

To what extent have the program resources been effective in the development of knowledge and skills for participant pharmacists?

Effectiveness

What have been the benefits/ strengths/weaknesses of the program resources?

In terms of effectiveness, how do DMAS program resources compare to other Fourth Agreement professional programs and services and meet requirements?



3.2 EVALUATION FINDINGS

The Program Resources utilised in stage 2 of DMAS were largely developed during Stage 1 of the program. As part of the development and planning of stage 2 the stage 1 program resources were modified and improved where appropriate. This development was done under the instruction of the Guidelines Consultant Group (GCG), which comprised the following members:

Joe Whitehouse (SA) - pharmacist

Bruce Elliot (QLD) – pharmacist, PSA Board member

Maree Simpson (NSW) - pharmacist

Cathy Beckhouse (ACT) – pharmacist, PSA Board member

Mark Coles (WA) – pharmacist, diabetes educator

Ines Krass (NSW) – pharmacist, University of Sydney

Jenny Bergin – pharmacist, Guild employee, PPSAC member

Sarah Gillespie – pharmacist, AACP employee

Jacquie Maycock – Department of Health and Ageing

Karen Farquhar – Department of Health and Ageing

Jim Hales - HOI

Paul Gysslink – pharmacist, APESMA

Chris Flood – pharmacist, Guild employee

The GCG group had representation from all partners, and a cross section of pharmacists and diabetes knowledge, and represented a suitable cross section of skills and stakeholders to undertake the development of the program resources.

Pharmacists spoke in a positive manner in relation to the program resources, in terms of appropriateness and effectiveness, and generally consider the quality of the program resources to be comparable to other Fourth Agreement professional programs and services.

Participant pharmacists were asked to consider a number of questions in relation to the program resources. In summary their views were:

78% (137) of survey respondents assessed the program resources as being effective, or very effective, in the development of their skills and knowledge (figure 3.1)

Figure 3.1: To what extent have the program resources been effective in the development of your knowledge and skills?



72% (127) of survey respondents assessed the Pharmacy Readiness Kit (PRK) as being effective, or very effective, in preparing for providing the DMAS program (figure 3.2)

Figure 3.2: How effective was the Pharmacy Readiness Kit (PRK) in meeting your needs in preparing for, and providing services?

41% (72) of survey respondents assessed the Clinical Resource Centre (CRC) as being effective or very effective in preparing them for providing the DMAS program (figure 3.3). Of particular note in this question is the fact that 71 respondents (40%) identified that they had not referred to the CRC. In our sub-analysis of these responses, and in our consultations, it was identified that some of the reasons for this was either not being successful in recruiting patients (13 of the 71, 18%), or not being aware that the CRC was available (9 of the 71, 13%).

Figure 3.3: How effective was the Clinical Resource Centre in meeting your needs in preparing for, and providing services

42% (73) of survey respondents assessed the Guild website as being effective in preparing for providing the DMAS program (figure 3.4). Of particular note in this question is the fact that 16% of respondents identified that they have not referred to the Guild website, and 35% (62 responses) that found it “somewhat” effective. Of those respondents that identified the website as somewhat effective, the following explanations were provided:

The website was difficult to find the information required, or they felt the need to supplement the information online by making a telephone call to the Guild (11 of the 62 responses, 17%), or



They felt there was too much duplication of information, either across sites, or having both online and hard copy resources (5 of the 62, 8%)

Figure 3.4: How effective was the Guild website in meeting your needs in preparing for, and providing services

Participating pharmacists (both surveyed and consulted) had varying views on the strengths and weaknesses of the program resources. However, the majority of opinion was that the program resources were comprehensive, well researched, well laid out and provided a very useful tool for pharmacists. The suite of pre-printed forms was identified as a key benefit.

Some pharmacists have presented opposing views suggesting that there was too much information, resulting in a bulky folder of materials that was difficult to access information quickly, particularly during a consultation with a patient. Some pharmacists identified the promotional materials as a point of weakness in the program resources, commenting that they were ineffective. The survey responses and our discussions with pharmacists have identified a number of opportunities for improvement, which have been summarised below in section 3.2.

3.3 OPPORTUNITIES FOR IMPROVEMENT

Survey responses and the consultations have identified a number of areas for improvement in relation to the program resources, which can be broadly categorised into:

Ease of use

Content – lifestyle & technical

Administration & other

3.3.1 EASE OF USE

A number of pharmacists commented that the PRK was too bulky to handle during consultations, and that due to the large amount of material it was sometimes difficult to know where to look or what to do next. The following suggestions for improvement may be worth investigating in future programs:

1. Have a single electronic source in relation to program resources and information rather than multiple sources. It was suggested that a single electronic or online source would be beneficial in reducing paper and could be readily updated. This electronic source would also incorporate all of the required program forms.

2. Including a more streamlined approach to processes, supplemented by the inclusion of decision trees or flowcharts, especially in relation to the following topics:

a) Patient management – what form to use next? (perhaps including a chronological timeline of when you would expect to use each form)



b) Patient management – step by step flowchart of your next patient appointment covering the key processes and the administrative elements required at each phase. This flowchart could also cover key consultation questions, but we recognise that each patient will have different needs, and may result in the flowchart remaining reasonably high-level.

c) How to recruit a patient

3. Redesign of the program resources table of contents may be of benefit. Some comments suggest the inclusion of a “program resources for dummies” page, or a more detailed index. An electronic program resource (suggestion 1 above) with a suitable search function or hyperlink may assist greatly in this regard. One suggested a complete redesign of the table of contents, suggesting the following as a more suitable flow of information:

a) Diabetic treatment guidelines

b) Forms

c) Working through the Patient Record

d) Blood Glucose Monitoring

e) Pattern Management

f) Resources for goal setting and review (strategies for lifestyle management, and strategies for medication adherence).

g) Extra resources/handouts

h) PRK at the end

4. A number suggested that a bullet-point approach with less preamble, as opposed to a text book approach would have resulted in a more concise product that was easier to navigate and would be easier to use.

3.3.2 CONTENT – LIFESTYLE & TECHNICAL

A number of useful suggestions have been made in relation to the content included in the program resources. Key areas identified were:

Pharmacists commented that they wanted more material in relation to diet and exercise. Our discussions with pharmacists identified that these matters were often key areas of focus and the inclusion on additional material for the pharmacists, but also materials for the patient would have been very useful.

1. The inclusion of a simple illustration on how the different diabetic medications work in the body. This could be used as a tool when explaining a medication's action with patients, for instance comparing Glucose Metabolism of a healthy person, in type 2 diabetes, and then the affect of different medications. A pharmacist identified the Healthworks Blood Glucose booklet (sponsored by Sanofi-Aventis) as containing something similar (but excluding the medication impacts).

2. Pharmacists identified that a comparison of BGM features and benefits of different makes and models would have been of great use, in both making recommendations to, and supporting, patients.

3. Inclusion of more case studies to assist in consolidating understanding.

3.3.3 ADMINISTRATION & OTHER

Key areas for improvement identified were:

1. Simplifying and streamlining the reporting of results. Some felt there was too much information required and some duplication.



2. The engagement of GPs was identified as a barrier by some pharmacists. Some suggestions to improve this in future programs were to include more literature in relation to patient/doctor management, and provide more information to GPs.

3. The inclusion of desktop tools (eg patient management software) in relation to patient management and consultations was suggested as a means to improve efficiency for pharmacists, as everything could be recorded online.

4. Some pharmacists suggested more promotional materials and options would have assisted in recruiting more patients to the program.

5. A DVD or video link of real life case study of someone doing the program with a patient & then steps for reporting, remuneration etc may help to consolidate training and provide a useful tool for pharmacists.



4

TRAINING DELIVERY

This chapter identifies our findings and areas for improvement in relation to the delivery format of the training.

Table 4.1 sets out the key questions presented in the Evaluation Framework in relation to the evaluation of the delivery format of the training.

Table 4.1: Delivery format of training evaluation questions

Evaluation Focus


Delivery format

What evidence exists to suggest a need for mandatory training, including pre-reading and credentialing requirement for the program?

Appropriateness

Was the delivery format for the training component of DMAS appropriate for its purpose?

Was an appropriate process and criteria used to select the “educators” appointed to deliver the workshops

Was the format used for the delivery of training the preferred format for pharmacists in preparing them to deliver the DMAS?

Effectiveness

What have been the benefits/ strengths/weaknesses of delivery format of the training components?


Similar to the program resources, the format of training delivery was essentially established in stage 1 of the program and modified and improved in stage 2. Our consultations identified that a number of improvements were made to improve the format of training delivery (based on stage 1 experiences). In addition an Expert Review Group (ERG) was established to oversee the development of the training and training materials. The ERG comprised the following members:

Mark Coles - pharmacist, diabetes educator

Ines Krass - pharmacist, University of Sydney

Kay Stewart – pharmacist, University of Melbourne

Greg Peterson – pharmacist, University of Tasmania, proprietor

Rachel Diennar – pharmacist from Tasmania

Jenny Bergin - pharmacist, Guild employee, PPSAC member



Sarah Gillespie - pharmacist, AACP employee

Jacquie Maycock – Department of Health and Ageing

Karen Farquhar – Department of Health and Ageing

Jim Hales – Health Outcomes International

The PSA managed the process of selecting educators to deliver the training in each state. The process involved advertising the roles, assessing applicant‟s credentials and ultimately appointing and scheduling educators to prepare for and present training. The process utilised for selecting educators is robust and in accordance with that generally utilised for the diligent appointment of personnel for contractual purposes.

Educators were provided with “trainer materials” to facilitate their preparation for training sessions. We understand from our stakeholder consultations that in some instances the educators received their training materials with little time to prepare for their session. Despite this the sessions received high effectiveness ratings, and further, the educators themselves were seen as knowledgeable in the subject matter by the vast majority of respondents (93%) as illustrated in figure 4.1 below. The “no” responses represent 12 pharmacists and further review of these respondents identified:

Four of the respondents attended training at Parkville

Three identified the need to include a Diabetes Educator (or similar) in the provision of the training

Figure 4.1: Did you consider the educators sufficiently knowledgeable in the subject matter to answer questions?

The delivery format was considered effective by participating pharmacists and by stakeholders. In our consultations the comments were generally very favourable in relation to the mix of pre-reading, workshops and case-studies. The survey results support the comments made in consultations with the following key findings:

Participants found each component of the training to be effective in developing skills and knowledge, with all three components being rated “effective” or better by more than 77% of respondents (table 4.2). The pre-workshop modules were rated most effective, with 88.1% of respondents rating them effective or better.



Table 4.2: Please rate the effectiveness of each component of the training program in the development of your knowledge and skills

Item Very effective Effective Somewhat effective

Not effective Total

Pre workshop modules: 38.1%

67 50.0%

88 10.2%

18 1.7%

3 176

Two-day workshop: 39.2%

69 44.9%

79 14.8%

26 1.1%

2 176

Case study: 23.9%

42 53.4%

94 21.6%

38 1.1%

2 176

Average % 33.7% 49.4% 15.5% 1.3% 176

Similarly, participants found each component of the training to be effective in preparing them for service delivery. All three components being rated “effective” or better by more than 68% of respondents (table 4.3), with the pre-workshop modules again being rated the most effective.

Table 4.3: Based on your experiences in the workplace, how effective was each component of the training program in preparing you for service delivery?

Item Very effective Effective Somewhat effective

Not effective Total


53 48.9%

86 18.2%

32 2.8%

5 176


55 44.9%

79 21.0%

37 2.8%

5 176

Case study: 21.0%

37 47.7%

84 23.3%

41 8.0%

14 176

Average % 27.5% 47.2% 20.8% 4.5% 176

Despite the pre-workshop format being rated in the survey as being effective, some of our preliminary consultations suggested that participants did not necessarily dedicate an appropriate amount of time to completing the pre-reading. We were advised that each of the four modules would most likely require four hours to complete – 16 hours in total. To assess this, a question was included in the survey and presented in figure 5.2 below in respect to the amount of time pharmacists spent on pre-reading. The responses suggest that the majority (73%) of pharmacist dedicated less than the 16 hours to the pre-reading materials, with 50.6% of respondents taking less than half the suggested time. Although this may have caused some frustration for educators in the two day workshops, it is unclear whether this has had a negative impact on the program itself (though some survey respondents did note this as a weakness, i.e. “some attending had not done the pre-reading”).



Figure 4.2: The delivery format included a pre-reading component, arranged into four

modules. Please provide an estimate of the amount of time you invested in completing the pre-reading.

Table 4.4 below summarises the survey responses of pharmacists with respect to identifying strengths and weakness of the delivery format (these are a selection of “representative” comments).

Table 4.4: Training delivery strentghs and weaknesses

Strengths Pre-reading modules

Hands on experiences with BGM

Educators were excellent

Delivered in an interesting manner

Weaknesses Need more case studies

Needed more role play based around patient interviewing

Would have been good to have more feedback on the case study (rather than just a pass/fail mark).

Two day workshops felt rushed, and had to revise all materials afterward to ensure it made sense

Long days with too much to cover

Needed more time on how to implement within the pharmacy in a practical sense

It is a shame to not have further face-to-face feedback with the trainers throughout the program, perhaps a 6-month follow-up session to revise, bring up issues and reiterate the important processes would be extremely helpful.

Have more questions at the end of sections




Many participants were unable to identify areas for improvement in relation to training delivery formats, however, the survey responses and the consultations have identified a number of areas for improvement which can be broadly summarised as:

1. To ease the amount of time required in travelling or spent in face to face sessions, some components of the course would have been well suited to an interactive online self study format. One that tested you periodically before proceeding to the next module. This may have been particularly useful for remotely located pharmacists.

2. Face to face time with a diabetes educator (rather than the DVD) would have provided greater benefit. Pharmacists noted that most patients had seen a diabetes educator, and it would be good to have done the same.

3. A focus on practical elements of implementing and running the program could have been achieved by including training elements such as:

a) More case studies

b) Role playing opportunities or patient scenario re-enactments. Effective patient consultation, one on one, is new to most pharmacists.



5

TRAINING MATERIALS

This chapter identifies our findings and areas for improvement in relation to training materials.

Table 5.1 sets out the key questions presented in the Evaluation Framework in relation to the evaluation of the training materials.

Table 5.1: Training materials evaluation questions

Evaluation Focus


Training Materials

Was an appropriate mechanism utilised and appropriate partners engaged in the development of the training materials?

Appropriateness

Did the training content include clinically appropriate material for Pharmacists delivering DMAS?

In terms of appropriateness, how do DMAS training materials compare to other Fourth Agreement professional programs and services?

To what extent have the training materials been effective in the development of knowledge and skills for participant pharmacists?

Effectiveness

In terms of effectiveness, how do DMAS training materials compare to other Fourth Agreement professional programs and services and meet training requirements?

Did the training materials adequately prepare pharmacists to complete the credentialing process?


Similar to previous sections, the training materials were essentially established in stage 1 of the program and modified and improved in stage 2. The ERG was established to oversee the development of the training and training materials.

The training materials were highly regarded by participant pharmacists, and this is demonstrated by the high “effectiveness” scores summarised in section 5. Furthermore, the successful credentialing of pharmacists provides further support of the training materials effectiveness (though first-time pass rates for the multiple choice component are low). The AACP have provided the following statistics in relation to pass rates:

Table 5.2: Credentialing pass rates

Pass first attempt Pass second attempt Pass third attempt

Short-answer test 100% NA NA

Hands on/BGM 100% NA NA

Multiple choice 14% 86% 0% (2 pharmacists)



The survey of pharmacists asked them to compare the training materials to other training programs they have attended. 21% of pharmacists considered the training materials to be of a higher standard than other training programs, and 44% rating the training materials as equivalent to other training programs (figure 5.1). We note that 22% reported that they were unable to compare as they had not attended other training programs. If we eliminate those who were unable to compare, or who had not attended other training, the results are: 30% rated as higher standard,

For those that responded that the materials were a lower standard, they cited the volume of paper and material as the explanation for their response.

Figure 5.1 In comparing the DMAS program to the other training programs you have attended, please compare the standard of the training materials to those programs?

In addition pharmacists were asked whether the training materials adequately prepare pharmacists to complete the credentialing process, with 168 respondents (95.5%) saying „yes‟.

Table 5.3 below summarises the survey responses of pharmacists with respect to identifying strengths and weakness of the delivery format (these are a selection of “representative” comments).

Table 5.3: Training materials strentghs and weaknesses

Strengths Very detailed

Videos especially the lifestyle module

Weaknesses needs to be more support for pattern management for pharmacists

There needed to be some information about coaching and motivation.


Many participants were unable to identify areas for improvement in relation to training materials, however, the survey responses and the consultations have identified a number of areas for improvement which can be broadly summarised as:

1. More emphasis on how to incorporate a profession service into a retail environment & overcoming resistance by customers who are time constrained (we note that this was the most commonly identified gap from respondent pharmacists that did not recruit any patients (20% of respondents with no patients recruited)).



2. More education in respect to building relationships with GP‟s and how to get them on board with the program.



6

CREDENTIALING

This chapter identifies our findings and areas for improvement in relation to the credentialing process.

Table 6.1 sets out the key questions presented in the Evaluation Framework in relation to the evaluation of the credentialing process.

Table 6.1: credentialing evaluation questions

Evaluation Focus


Credentialing Process

Was the credentialing process an appropriate test of pharmacists acquired knowledge? Appropriateness

What evidence exists to show the need for continuing professional development and renewal of credentialing?


The credentialing process was managed by the AACP. In our consultations with stakeholders it was identified that there were some difficulties in finalising the credentialing process as the training content was still being written very close to the point of commencement of the training sessions. To a certain extent, we understand that training content and credentialing materials were being prepared concurrently, rather than successively, as would ideally have been the case.

However, as noted in previous sections, pharmacists felt adequately prepared for the credentialing process. Further, they were asked to assess the appropriateness of each component of the credentialing process. There results are presented in table 6.2 below.

Table 6.2: The appropriateness of each component of the credentialing process in respect to ensuring a pharmacist is adequately prepared/trained to provide

DMAS:

Item More than appropriate

Appropriate Somewhat appropriate

Not appropriate

Total

Multiple choice questions: 24.4%

43 68.8%

121 6.3%

11 0.6%

1 176

Short answer questions: 25.0%

44 69.9%

123 5.1%

9 176

BGM assessment: 21.0%

37 65.3%

115 12.5%

22 1.1%

2 176

Average % 23.5% 68.0% 8.0% 0.6% 176



Pharmacists were surveyed in relation to their views on the future training and credentialing requirements with the following findings:

Pharmacists believe that there is a need to continue their professional development in relation to DMAS. Many felt this could be achieved through online self-study, or by incorporating a DMAS update into broader educational conferences or newsletters.

69% of pharmacists thought the re-credentialing was necessary and the PSA and the AACP were also of the view that re-credentialing is necessary.

Pharmacists were asked their view on appropriate re-credentialing timeframes, with every 2 years proving the most common response (50% of responses) (figure 6.1 below). The AACP consider that are-credentialing period of 1 to 2 years would be appropriate, but also suggested that it could be incorporated into annual professional development standards and requirements.

Figure 6.1: How regularly do you think the re-credentialing process should occur?


The key areas for improvement in relation to credentialing are as follows:

1. The short answer test papers were forwarded to the AACP at the conclusion of each two day training session. There were some significant delays in these being forwarded by the trainers to the AACP for assessment (despite there being a short deadline set for the delivery of papers). An online short-answer module would have eliminated the need for papers to be physically retained and posted.

2. The re-testing of participants that failed multiple choice questions was co-ordinated in a labour intensive manner. The re-testing process could have been made more efficient by automating the question re-selection process.

3. The timing of the multiple choice questions (testing participant‟s knowledge of the pre-reading material) was completed after the two day workshop. Given that section 4 identified some concerns about the completion of pre-reading, it may have been more appropriate to test participant‟s knowledge prior to being accepted into a face to face training session.

4. Some surveyed participants suggested that credit should be given for pre-existing competencies and qualifications. To that end, it may be useful to consider program and credentialing structures in future programs to accommodate this suggestion.



7

CLINICAL APPROPRIATENESS

A clinical expert team was appointed by HOI to consider certain aspect of the evaluation. In particular they were asked to provide responses to the questions presented in the table below:

Evaluation Focus Key research questions Question for clinical expert

Program Resources

Was an appropriate mechanism utilised and appropriate partners engaged in the development of the program resources?

Were the qualifications of the program resources development team suitable to ensure the program resources were clinically sound??

Training Materials

Was an appropriate mechanism utilised and appropriate partners engaged in the development of the training materials?

Were the qualifications of the training materials development team suitable to ensure the program resources were clinically sound?



Was the delivery format for the training component of DMAS appropriate for its purpose?

Was the delivery format for the training component of DMAS (ie pre-reading modules, two-day workshop) appropriate to educate pharmacists in DMAS?

Credentialing Process

Was the credentialing process an appropriate test of pharmacists acquired knowledge?

Was the credentialing process an appropriate test of pharmacists acquired knowledge?

All focus areas

What are the gaps and opportunities for improving:

Program resources

Training materials

The delivery format of the training component

The credentialing process?

What are the gaps and opportunities for improving:

Program resources

Training materials

The delivery format of the training component

The credentialing process?

7.1 OVERALL CONCLUSIONS FROM A CLINICAL PERSPECTIVE

The entire DMAS resources, material, delivery process and credentialing is well designed to give pharmacists adequate knowledge and enable them to provide the service with confidence. It could be made more efficient with the improvements suggested below.




7.2.1 QUALIFICATIONS OF THE PROGRAM RESOURCES DE VELOPMENT TEAM

In order to assess the appropriateness of the qualifications of the program resources development team, the program resources were reviewed to determine their purpose and content. The Pharmacy Readiness Kit was identified as the main collection of program resources and contains the operations manual, pharmacy forms toolkit, professional guidelines and standards, training and credentialing, promotional materials and evaluation information. The key stakeholders for DMAS, in particular users of the program resources tool, were also considered.

The program resource development team consisted of thirteen individuals including ten pharmacists (one of whom was also a diabetes educator), two employees from the Department of Health and Ageing and one employee of Health Outcomes International. The information made available for the assessment of the appropriateness of the qualifications of this team was limited to their name, profession, state or territory of practice and relevant affiliations with organisations. The educational qualifications and experience of the individuals was not disclosed and thus limited the scope of answering the question. Thus, we expanded this question to consider not only the qualifications (ie. the profession) of the individuals but also the location of individuals and relevant organisational affiliations.

Given this program was developed to be delivered by community pharmacists in a community pharmacy and the resources were primarily in place to facilitate the implementation of this service, the inclusion of pharmacists in the resource team was very appropriate. Although the information provided did not disclose the setting that the pharmacists on the team worked in or their experience, it is assumed that at least a proportion of these pharmacists worked in the community pharmacy setting and/or had experience in working as a community pharmacist. As this program was rolled out nationally, it was also appropriate that at least 5 states and territories had a pharmacist representing their region on the team.

It was also suitable that representatives from the three key organisations driving the development and delivery of the service were included on the team. The Department of Health and Ageing, the Pharmaceutical Society of Australia and the Australian Association of Consultant Pharmacists were all represented. In addition, a member of the Professional Programs and Services Advisory Committee was also on the team which was suitable to provide information and guidance from the general committee.

Whilst the DMAS program is a community pharmacist led program, it is designed to be delivered in collaboration with General Practitioners. Thus, a key area lacking in the program resources development team was representation of General Practitioners. This could have been accomplished by inclusion of a General Practitioner, a representative from the Divisions of General Practice and/or a representative from the Royal Australian College of General Practitioners. In particular, their inclusion would have been appropriate in the development of information regarding DMAS to be distributed to local health professionals, the development of professional standards and guidelines for DMAS and the operations manual, particularly around the coordination of the roles of General Practitioners and community pharmacists. This also may have led to an increased awareness of DMAS amongst General Practitioners.

Consumer representation was also absent from the resource materials development team. As a key stakeholder and participant in this program, consumer representation could have informed the development of consumer resources for DMAS and also provided feedback on any consumer-related content within the program resources. Consumer representation could have been sought through inclusion of a member of Diabetes Australia.

In conclusion, there is nothing to suggest that the qualifications of the program resources development team are not suitable to ensure the program resources are clinically sound. We would however recommend that future training programs for similar services consider the inclusion of representatives of other key stakeholders, such as General Practitioners and consumers, on the program resources development team.



7.2.2 QUALIFICATIONS OF THE TRAINING MATERIALS DEVELOPMENT TEA M

In order to assess the appropriateness of the qualifications of the training materials development team, the training materials were reviewed. The training materials included presentations from four pre-workshop modules and from the two-day workshop.

The training materials development team was comprised of ten individuals including seven pharmacists, two employees of the Department of Health and Ageing and one employee of Health Outcomes International. Again, limited information was supplied regarding the qualifications and experience of the team members.

Given the training materials were developed for pharmacists, it was appropriate that a large proportion of team members were pharmacists. Of the pharmacists, 3 were affiliated with universities which were assumed to indicate experience in academia. This was ideal given the need for people with teaching experience to be involved in the development of the training materials. In addition, one of the pharmacists was also a diabetes educator which may have enabled them to contribute knowledge as a professional in diabetes education. One pharmacist was a Guild employee who was suitable to provide representation of community pharmacists and in particular, community pharmacy proprietors. Another pharmacist was an employee of the Australian Association of Consultant Pharmacists which was vital given that this association was to train and accredit the pharmacists to provide the service.

The inclusion of one or more person with medical qualifications would have been ideal for the clinical training development team given the clinical nature of some of the material presented. In particular, we feel that a general practitioner and an endocrinologist would have been valuable in ensuring the material presented was up-to-date, accurate and clinically relevant from a medical perspective. Should this not be achievable, it would be advisable that general practitioners and endocrinologists are involved in reviewing the content of the training materials. It is not clear whether or not the training materials development team consulted doctors regarding the content of the training materials throughout their development.

In conclusion, there is nothing to suggest that the qualifications of the program resources development team were not suitable to ensure the training materials are clinically sound. We would however recommend that future training programs for similar services consider the inclusion of appropriate medical practitioners in the training materials development team, in this case an endocrinologist and a general practitioner, or at a minimum get the training materials reviewed by appropriate authorities like college of physicians or college of general practitioners.

7.2.3 THE TRAINING CONTENT

The training of pharmacists delivering DMAS was presented via four pre-workshop modules and a two day in-person workshop. The majority of the clinical content was covered in the four pre-workshop modules, each consisting of reading material, a video recorded presentation and case studies.

Module 1 had a video presentation by Prof Donald J Chisholm of the Garvan Institute, a renowned expert on Diabetes, discussing the prevalence of diabetes in Australian, American and other ethnic populations. This was followed by topics on genetic predisposition, prevalence of cardiovascular risk factors in diabetic population, metabolic syndrome, Insulin sensitivity, Insulin resistance, central obesity, mechanism of action of insulin, beta cell failure. This video was complemented with a manual developed by Pharmaceutical Society of Australia (PSA) which covered the above topics plus symptoms, clinical features, causes, risk factors for different types of diabetes. Apart from this endocrine regulation of glucose metabolism and differences between type 1 and type 2 diabetes were presented. We believe that this module gave pharmacists sufficient clinical background on the pathophysiology and epidemiology of diabetes to ensure they can explain these issues to patients enrolled in DMAS.

Module 2 elaborated on acute complications related to diabetes (eg. hypoglycaemia and hyperosmolar hyperglycaemic non-ketotic syndrome) and included information regarding the signs, clinical features and management of such clinical presentations. The manual also covered topics on diagnosis, mechanism, management of microvascular complications retinopathy, nephropathy and neuropathy. Macrovascular complications were also discussed. Whilst this module was very relevant to pharmacists providing the



service, some of the information regarding medication management included medications which are not available in Australia.

For example, the table of medications for management of diabetic neuropathy lists venlafaxine 75mg TDS which implies the normal release formulation however only modified release capsules are available in Australia meaning that it has to be a single daily dose. Another product listed was fluphenazine 0.5mg tablets which is only available as a depot injection in Australia.

Customising the information to the Australian setting would make the training more relevant for the pharmacists and reduce the potential for confusion. Resources such as the National Prescribing Service and the Therapeutic Guidelines may be a useful resource for Australian guidelines.

Module 3 had a video presentation by Prof Donald J Crisholm covering some landmark clinical trials UKPDS, DCCT, HPS, ACCORD, ADVANCE and their key findings in context of diabetes. The manual developed by PSA described the pharmacotherapy for diabetes including biguanides, sulphonyl ureas, thiazolidinediones, alpha glucosidase inhibitors, dipeptidyl peptidase-IV inhibitors, Incretin mimetics (like Exenatide) and various types of Insulins. The relevant mechanism of action, clinical effects, adverse drug reactions and dosage was covered in the manual for the oral antidiabetic medications. Apart from this weight loss medication, antiplatelet agents and pharmacotherapy of macrovascular risk factors like hypertension and dyslipademia were covered. This module gave pharmacists a therapeutic update on medications used to treat diabetes which is particularly important given the new agents and formulations which have been introduced over the recent years. Incorporating medications for the treatment of related risk factors ensured this module was comprehensive. We suggest that some of the landmark trials that are mentioned in the video presentation (e.g. UKPDS, ADVANCE, ACCORD) be included as part of the complementary pre-workshop module reading materials. By doing so this would allow easy access of these studies to the participating pharmacists and also provide a background to them about the evidence behind the guidelines. Apart from that, we also suggest that the role of some of the newer agents (e.g. sitagliptin) in the step-wise management of type 2 diabetes to also be included so that the participants understand the roles of these medications.

Module 4 comprised of a video on lifestyle management Dr Jane Overland and a manual developed by the PSA covering the lifestyle issues of nutrition and glycaemic Index, exercise, weight loss, alcohol consumption and smoking. The in-person 2 day workshop covered topics on patterns of Blood Sugar Level (BSL) management using pharmacotherapeutic interventions such as oral antidiabetic medications and insulin. The physiological response to insulin, algorithms to establish insulin requirements and various types of insulin regimes to cater for the basal and bolus requirements were presented along with simultaneous use of anti diabetic medications. An interactive case study was employed to enable pharmacists to understand how this information applies in practice. In addition, the pharmacists were given the opportunity to explore, understand and use various BSL meters like Optium Exceed, Freestyle Lite, Care Sens, Accu-chek Go, Integra, Performa and different software programs used to analyse results from these meters. The workshop also included a session on the use of different insulin delivery devices. These sessions were particularly clinically appropriate given the need for pharmacists delivering this service to be able to interpret such information and be able to instruct on the use of such devices. Other non-clinical information regarding the day-to-day operations and delivery of DMAS were also included in the two-day workshop.

Overall, the topics covered in pre-workshop modules and the 2 day workshop covered relevant and current information on diabetes and as such was clinically appropriate for pharmacists delivering DMAS. Our main suggestion is to ensure that the information delivered is relevant to the Australian setting as inclusion of medications and guidelines from overseas can cause ambiguity.

7.2.4 THE DELIVERY FORMAT

The training component of DMAS consisted of four self-directed online pre-workshop modules and a two-day face-to-face workshop delivered by workshop trainers. Each of the pre-workshop modules covered a topic related to type 2 diabetes and was presented in the form of a training manual, video presentation (except for Module 2) and sample multiple choice questions. These were all available online.



However, the only way for the participating pharmacists to access the modules prior to workshops was via the internet. This could potentially be an issue for a busy community pharmacist who does not have time to complete the training modules at work and also does not have internet access after work. Also, we suggest the format of the four preworkshop modules to be delivered in a consistent manner. In particular a video presentation is missing from Module 2. We suggest that a video presentation about the complications of diabetes from an expert would be useful for the participating pharmacists.

Within the training manual, the learning objectives, „quick review‟, practical knowledge application and module review are outlined. This allowed the participants to first have a general overview of what is being expected, and then while going through the module, they were able to reflect on the clinical knowledge they were learning. „Quick review‟ comprised of questions that are related to a sub-topic and is presented in the format of true/false, objective questions and „fill in the blank‟ type questions. We think that the role of „quick review‟ is to allow the pharmacist to reflect on their own knowledge gap before they move on further. We suggest that these questions be delivered in a straightforward manner, in particular just true/false, and objective questions. We think that some of the „fill in the blank‟ type questions may only serve to encourage the participants to memorise the training materials. The section on practical knowledge application comprised of two case studies. These case studies allowed the participants to apply the key knowledge that would be important when implementing DMAS. The case studies from Module 1 to Module 4 are related and reflect the disease progression of a patient that a community pharmacist may encounter. Pharmacists were encouraged to bring in their answers to the pre-workshop module case studies to the workshop with the correct answers provided on the workshop days. While we are unsure of any open opportunity on the workshop days that these case studies are being discussed, we assume that there would be discussions to clarify some of the questions that the participating pharmacists might have. This is also beneficial for the pharmacists to reflect on any knowledge gaps that they might have identified in this process. The module review was presented in the form of objective questions and aimed to ensure that the participating pharmacist meets the learning objectives. The participants were required to complete sample multiple choice questions before the next module was released. We think this is ideal and ensures that they have fully understood one module before moving on to the next.

The two-day workshops focused on the documentation, setting up and actual implementation of the DMAS program in the participating community pharmacy. The workshops were interactive and covered the importance of the DMAS program and how to implement it, how to store data for future reference, interviewing skills and practical sessions including how different blood glucose monitors work. We think that the delivery format of the education and the assessment of the practical skills is adequate on the workshop days. These provided opportunities for the participating pharmacists to have hands-on practice of what they would possibly encounter when they implement DMAS in their pharmacies. At the workshop a hardcopy of the training materials was provided to the participating pharmacists and a DVD was posted out later. This DVD includes all the materials that the participating pharmacists already had accessed. We suggest that the training materials (a hardcopy and the DVD) be provided to the participating pharmacist prior to the workshops, including the outlines and the objectives of the workshops.

We think that this delivery format is important as this can cater for the participants who do not have access to internet outside work, and also those who may wish to review, identify gaps and prepare themselves before the workshops. Apart from that, since the workshops focus more on hands-on practice we think that by delivering the training materials before the workshops would not defeat or compromise the purpose of the workshops. Overall, since the participants‟ understanding of the disease and the practical management of BGL monitoring are being assessed by AACP, we think that providing the training materials prior to the workshop would not compromise the participant‟s understanding. After the completion of the workshops, follow up training of BGL monitoring or settings in individual‟s pharmacy can be provided by AACP upon individual‟s request. We think that this is very positive, given that some participants may encounter issues when implementing DMAS in their pharmacies. In keeping with this, we feel that pharmacists may also benefit from the establishment of an informal support group. This would provide the opportunity for the participating pharmacist to ask clinical questions regarding the disease or seek advice regarding the difficulties that they encounter when they implement DMAS. This could be delivered in the form of an online discussion forum or as a working group meeting. This form of



support group would also be useful for participating pharmacists who wish to seek advice or help from other participating pharmacists after they have completed the trainings.

In conclusion, from the information provided to us regarding the delivery of the DMAS training, we feel that the current delivery format of the training components of DMAS is appropriate to educate the pharmacists to deliver DMAS. However consideration should be given to providing hardcopy and electronic versions of the training materials prior to the workshops. Access to continued training if required may also need to be considered.

7.2.5 THE CREDENTIALING PROCESS

The Australian Association of Consultant Pharmacy (AACP) was responsible for the assessment and credentialing of the participating pharmacists against the DMAS competency based training course. AACP has always been involved in the accreditation and support of the consultant pharmacists who wish to conduct Home Medicines Review or Residential Medication Management Review in Australia. Hence it was appropriate for AACP to be the key responsible organisation for the credentialing of the participating pharmacists in DMAS.

The credentialing process consisted of both practical and theoretical components evaluating the two aspects required for efficient delivery of any professional service. The assessment process was delivered in the form of multiple choice questions, short answer questions, and hands-on demonstration of blood glucose meters and related software. The contents of the credentialing process included three major components: assessment of clinical knowledge of type 2 diabetes and its management (theoretical assessment), assessment of blood glucose meters and software (practical assessment), and assessment of interpretation of blood glucose readings and communication with patients (theoretical assessment).

The assessment of the clinical knowledge related to type 2 diabetes was delivered in the format of twelve Multiple Choice Questions (MCQs), with three questions based on each of the four pre-workshop modules. These MCQs were available on the first business day following the completion of the 2-day workshop and were required to be completed within seven days. A minimum pass mark of 68% i.e. 8 correct answers out of 12 is required.

The first module was based on the epidemiology and pathophysiology of type 2 diabetes. The related MCQs reflect a good combination of questions for assessing competency in these areas. The content of the questions included insulin deficiency, insulin resistance, complications, risk factors, HbA1c and various types of diabetes. However there are no questions related to the new concepts in the pathophysiology of type 2 diabetes like incretin hormones and DPP IV inhibitors. We think it is important that the participating pharmacists to be kept up-to-date and hence suggest it would be useful for the new concepts and their role in type 2 diabetes to be incorporated into the questions.

The second module was based on the complications of type 2 diabetes. The microvascular complications of type 2 diabetes are covered very well. However, it would be ideal should more questions related to macrovascular complications and cardiovascular risk calculations be included in the assessment.

The third module was based on the pharmacotherapy of type 2 diabetes. The types of questions that are related to this module reflect knowledge required from the fundamentals to the latest developments in the pharmacological treatment of type 2 diabetes. Mechanism of action, side effects, dose reductions and drug interactions are included. We suggest assessment questions on insulin treatment options also be included.

The fourth module was related to lifestyle management of type 2 diabetes. All the relevant counselling aspects of lifestyle management including diet, exercise, alcohol intake and smoking cessation with practical examples are included.

The second component of the credentialing process included practical assessment of blood glucose meters and related software. This practical assessment was carried out in pairs, with one participating pharmacist assessing another prior to swapping over. A checklist developed by AACP was provided to the observing participating pharmacist. The observing participating pharmacist assessed the other by checking against the checklist. This paired assessment was supervised by a trainer. The checklist developed by AACP incorporated all the aspects of operational procedures for various blood glucose meters and their related softwares. We suggest assessment on the usage of different insulin pen devices be added to the



credentialing process, if possible. Ideally this process could be assessed by a credentialed diabetes educator.

The third component of the credentialing process was a theoretical assessment and comprised three short answer questions related to the interpretation of blood glucose readings and effective communication with patient. This was an open book assessment where the training manual and any other relevant reference materials could be used. The short answer questions were marked on a 5-point bipolar semantic differential scale (appropriate-inappropriate). A pass mark is midline or better on all questions and formative feedback will be provided to those who do not meet the requirements. The assessment included three aspects of diabetes: appropriate interpretation of the blood glucose readings and trends, appropriate medication management and/or lifestyle advice to patient and/or GP, and effective communication and management of adherence issues with patients. In particular, these aspects included an understanding of the trends in blood glucose readings and identifying possible reasons for abnormal results based on history or medication and advise accordingly, post-prandial hyperglycaemia, early morning hypoglycaemia, behavioural modification, goal setting and education. It is suggested that the cases could be made slightly more complex and closer to real practice by incorporating the insulin component into them. This could enable the participating pharmacist to be better equipped with various situations that they may encounter in their practising pharmacy.

Formative feedback was provided to the pharmacists who do not meet the credentialing requirements as compared to the summative feedback which only provides a mark. Formative feedback is useful resource to reflect on any learning and knowledge gaps and to rectify them.

With the MCQs, the participating pharmacists were required to redo only the same question(s) that they answered incorrectly. This allowed them to focus on the questions that they answered incorrectly, but also allowed them to be more familiar with the questions for re-assessment.

In conclusion, we think that the overall credentialing process is well designed to assess the participating pharmacist‟s acquired knowledge but at the same time with few minor improvements, as suggested above, could make it more efficient and increase the pharmacist‟s confidence in providing the service.

7.2.6 GAPS AND OPPORTUNITIE S FOR IMPROVEMENT

The key areas for improvement in relation to the clinical review are as follows:

1. An apparent gap in the program resources is a link between community pharmacists and GPs. Given this, we feel that it would be resourceful to have a facilitator(s) between the GPs and community pharmacists. The role of the facilitator would be to raise awareness of DMAS to GPs, and to highlight the roles of GPs in the service. This would be particularly advantageous in the initial phases of the program.

2. In relation to the content of the training materials, it would have been clearer and apt if the clinical information was customized for the Australian setting, as opposed to inclusion of information related to overseas countries. This would better suit the needs of the community pharmacists delivering DMAS. In addition, we feel that Australian cardiovascular risk calculators/charts and their interpretation could be introduced to the pharmacists during the pre-workshop modules or workshops. This would provide pharmacists with the skills and tools to provide a more comprehensive service to these patients who are generally at high cardiovascular risk and make more informed decisions and recommendations surrounding pharmacotherapy and lifestyle changes.

3. Regarding the presentation format of the training materials, we suggest reviewing access to the pre-reading modules entirely through the internet to cater for the needs of pharmacists who may have trouble accessing the internet outside of working hours. Also, it would be consistent and useful to have a video presentation for all of the pre-reading modules, thus ensuring an equal amount of information for each module is presented. We also suggest providing the hard copies and video presentations of the material along with the online modules prior to the workshop. Provision of key studies related to diabetes management prior to the workshop should also be considered.



4. It is important that there is ample opportunity to ask questions and clarify information in regards to the pre-workshop modules on the workshop days.

5. We suggest a process be put in place for continued support and training at the pharmacy level, if required.

6. The credentialing process could be made more comprehensive by incorporating more MCQs or case studies as part of the credentialing process in addition to those in the pre-reading modules. Also, assessment of insulin administration devices by a diabetes educator would be useful to ensure for the pharmacists are competent in this area (though we recognise this could be logistically difficult and would have had cost implications for the program). The graphic interpretation of the blood glucose readings could be made more closely related to the real practice and enable pharmacists to deal with any kind of situation if a case study with insulin is included in the credentialing process. We feel that the section on graph interpretation should be extended to take into account insulin usage, as the case study only illustrates the use of oral hypoglycaemic agents. Many patients use insulin and hence the graph and its interpretation would be different.



8

OTHER FINDINGS

This section summarises other findings in relation to our consultations.

8.1 COST EFFECTIVENESS

The PSA‟s and AACP‟s training and credentialing budgets were managed through subcontracts administered by the PGA. Our discussions with stakeholders identified that these budgets were sufficient to fund the training and credentialing process. We have been unable to provide a state by state analysis of the costs of training delivery as the financial information has not been collated in that manner.

Our review of financial reports of the AACP and PSA confirm that allocated funds have been expended (PSA financial reports indicate that the training components were overspent, offset by an underspend in “clinical implementation support”).

8.2 WITHDRAWING PHARMACISTS

As identified in the Evaluation Framework, we consulted with five pharmacists that withdrew from the DMAS program to determine if the training was a factor in their decision to withdraw. We have determined that this was not the case, and that the withdrawing pharmacist spoke favourably of the training. Their views were consistent with the survey results presented earlier in this report.

Withdrawing pharmacists identified a lack of time to recruit patients, or difficulties in engaging with GPs as their reason for withdrawing. Despite identifying these two factors as reasons for withdrawing, the pharmacists consulted did not identify any areas for improvement (in relation to the training) to address patient recruitment or GP engagement.

8.3 POST TRAINING & IMPLEMENTATION SUPPORT/MENTORS

A number of pharmacists suggested that some post-training support would have been invaluable. Various mechanism were suggested, ranging from receiving a follow-up phone call at key stages post training, to the inclusion of a formal mentor program, where they could call an appropriately experienced person for assistance when necessary.

We believe that this concept would be worthy of consideration in future programs.

8.4 TRAINING SUBSIDY

A training subsidy was available for participating pharmacists under the program, to cover certain defined costs in relation to travel, and backfill of staff while in attendance at the two day workshop. To assess the perceived benefit of the subsidy, three questions were presented in the online survey, and in our telephone consultations in relation to the subsidy. The findings were:

84.6% of survey respondents applied for the subsidy,

75.6% of survey respondents assessed the subsidy as sufficient to cover the costs of attendance, and



61.9% of survey respondents identified that the availability of a subsidy was a significant or critical factor in their decision to participate in the program (refer diagram 8.1 below). However, in our discussions with pharmacists, it was evident that they were very enthusiastic about the opportunity to develop skills and provide services in relation to DMAS, and the availability of the subsidy made for a simpler discussion with their employer when requesting to attend the program.

Figure 8.1: What value did you place on the availability of a training subsidy when making your decision to participate in the DMAS?

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-I

A APPENDIX A – QUANTITATIVE SURVEY

RESULTS

1. HOW MAN Y PA TI EN TS HA V E YOU R EC RUI TED U NDE R T HE DMAS?

Item Count Percent %

2 to 5 79 44.89%

more than 5 39 22.16%

None 30 17.05%

1 28 15.91%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-II

2. DID YOU APPLY FOR THE TRA IN I NG SUBSIDY A VAI L ABLE UND ER TH E PROG RA M?


Yes 148 84.57%

No 27 15.43%

3. HOW EFFECT IV E WAS TH E TRA I NI NG PROGRA M I N P REPA RI NG YOU TO PRO V IDE THE DMAS IN YOUR

WORKPLA CE?


Effective 76 43.18%

Very effective 60 34.09%

Somewhat effective 38 21.59%

Not effective 2 1.14%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-III

4. HAVE YOU PA RT IC IP ATED IN AN Y PR EV IOU S TRA I N I NG PROGR AMS MA NAGE D BY THE

PHAR MAC EUT IC AL SOC IET Y O F AUSTRALIA , AUSTRAL IA N ASSOCI AT ION OF CONSUL TAN T PHA RMACY O R

THE PHAR MA CY GUILD OF AUSTRAL IA?


Yes 121 68.75%

No 55 31.25%

5. IN CO MPAR ING THE DMAS PROGRAM TO TH E OT HER T RA IN I NG PROGRA MS YO U HAVE A TTE NDED

(ABOV E), HOW EFFE CT IV E WAS TH E T RAI N ING PROGRA M I N PR EPAR I NG YOU TO P R OVIDE TH E DMAS IN

YOUR WORKPLA CE?


Equivalent to other training programs 71 40.34%

I have not participated in other training programs 39 22.16%

More effective than other training programs 36 20.45%

Less effective than other training programs 16 9.09%

Unable to compare 14 7.95%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-IV

6. THE DELI VE RY FORMAT I N CLUDED A PR E-READ ING COMPON E NT , AR RA NGED I NTO FOUR M ODUL ES .

PLEASE P ROV IDE A N ES T I MAT E OF TH E AMOU NT OF TI ME YOU IN VES TED IN C OMPLE TI NG TH E PR E-READ ING .


5 to 10 hours 58 32.95%

11 to 15 hours 40 22.73%

2 to 4 hours 31 17.61%

16 to 20 hours 27 15.34%

More than 20 hours 17 9.66%

Less than an hour 3 1.70%

7. PLEASE RA TE TH E EFFEC T I VE NESS OF EA CH COM PONE NT OF TH E T RA IN I N G PROGRA M IN TH E

DEVE LOPMEN T OF YOU R K NOWLEDG E A ND SK ILLS :

Item Very effective Effective Somewhat effective Not effective Total


67 50.0%

88 10.2%

18 1.7%

3 176


69 44.9%

79 14.8%

26 1.1%

2 176

Case study: 23.9%

42 53.4%

94 21.6%

38 1.1%

2 176

Average % 33.7% 49.4% 15.5% 1.3%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-V

8. BASED ON YOUR E XPER I E N CES I N THE WO R KPLAC E , HOW EFFE CT IV E WAS EA CH COMPO NE N T OF T HE

TRA IN I NG PROG RAM I N PR E PARI NG YOU FOR S E RV I CE DELI VE RY?

Item Very effective Effective Somewhat effective Not effective Total


53 48.9%

86 18.2%

32 2.8%

5 176


55 44.9%

79 21.0%

37 2.8%

5 176

Case study: 21.0%

37 47.7%

84 23.3%

41 8.0%

14 176

Average % 27.5% 47.2% 20.8% 4.5%

9. DID YOU CONSID ER T H E EDUCA TORS SUFFIC IE N T L Y KNOWL EDGEA BLE IN THE SUBJEC T MA TT ER T O

ANSWER QUEST IO NS?


Yes 164 93.18%

No 12 6.82%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-VI

10. IN COMPA RI NG TH E DMAS PROGRAM TO THE OT HER T RA IN I NG PROGRA MS YO U HAVE AT TE NDED

(Q6 ABOVE), PL EASE CO MP ARE TH E STA ND A RD OF T HE TRA IN I NG MAT ER IALS TO TH OSE P ROGRA MS?


Equivalent to other training programs 78 44.32%

I have not participated in other training programs 38 21.59%

Higher standard than other training programs 37 21.02%

Unable to compare 13 7.39%

Lower standard than other training programs 10 5.68%

11. DID TH E TRA I NI NG ADEQ UATE LY PR EPA RE Y O U TO CO MPL ETE TH E C R E DENT IAL I NG PRO CESS

(MUL TIPL E CHO IC E , SHOR T ANSWER QUEST IO NS A ND BGM ASSESSME NT)?


Yes 168 95.45%

No 8 4.55%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-VII

12. TO WHAT E XT EN T HA VE T HE PROG RA M RESOU RC ES BEE N EFFEC TI VE I N TH E DE VELOP ME NT OF YOU R

KNOWL EDGE AND SKILLS?


Effective 82 46.59%




13. HOW EFFE CT IV E WAS THE PHA RMACY RE ADI NE SS K IT (PRK) I N MEE TI NG YOUR N EED S I N

PREPA RI NG FOR , AND PRO V IDI NG SE RV I C ES?

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-VIII


Effective 87 49.43%



I have not referred to the PRK 7 3.98%


14. HOW EFFEC TI VE WAS THE CL IN I CAL RESOURC E CEN TR E I N MEET I NG YOU R N EEDS IN P REP AR ING

FOR , AND PROV IDI NG S ER VI C ES


I have not referred to the Clinical Resource Centre 71 40.34%

Effective 50 28.41%




The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-IX

15. HOW EFFE CT IV E WAS THE GU ILD WEBS ITE I N MEET I NG YOUR N EEDS I N P REPA RI NG FOR , AND

PROVID I NG SE RV IC ES



Effective 56 31.82%

I have not referred to the Clinical Resource Centre 29 16.48%



16. PLEASE RAT E THE APP RO PRI ATE N ESS OF EACH C OMPON EN T OF THE CR ED E N TIAL I NG PROC ESS IN

RESPE CT TO ENSU RI NG A PH ARMA C IST IS AD EQ UAT ELY P REPA RED/TRA I NED TO PRO V IDE DMAS:

Item More than

appropriate Appropriate

Somewhat

appropriate

Not

appropriate Total

Multiple choice questions: 24.4%

43 68.8%

121 6.3%

11 0.6%

1 176

Short answer questions: 25.0%

44 69.9%

123 5.1%

9 176

BGM assessment: 21.0%

37 65.3%

115 12.5%

22 1.1%

2 176

Average % 23.5% 68.0% 8.0% 0.6%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-X

17. DO YOU THI NK PHAR MAC I STS SHOULD BE P ER I OD ICALL Y R E-C RED EN TI ALE D IN REL A T ION TO

PROVID I NG A S ER VI CE SUC H AS T HE DMAS?


Yes 123 69.89%

No 53 30.11%

18. HOW R EGULARL Y DO YO U TH INK TH E RE-CR EDE NT IA LING PRO CES S SHOULD OCC UR?


Every 2 years 88 50.00%

I do not think that there is a need for re-credentialing 38 21.59%

Every 4 years 32 18.18%

Every year 18 10.23%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-XI

19. HOW APPROPR IAT E WAS THE SCH EDULI NG A ND AVAI LAB IL ITY OF COURS ES UNDE R T HE DMAS

PROGRAM?


Appropriate 114 64.77%

Somewhat appropriate 33 18.75%

More than appropriate 21 11.93%

Not appropriate 8 4.55%

20. WHAT VALUE DID YOU PL ACE ON TH E AVA ILA BI L I T Y OF A TR AI NI NG SU BSIDY WHE N MAK I NG YO UR

DECIS ION TO PAR TI CIP ATE IN TH E DMAS?


Significant factor in my decision 67 38.07%

Minor factor in my decision 45 25.57%

Critical in my decision 42 23.86%

Not a factor in my decision 22 12.50%

The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage 2 of the Diabetes Medication Assistance Service Evaluation Report June 2010 A-XII

21. WAS THE T RAI N ING SUBS IDY AMOU NT SUFFIC IE N T TO CO VE R T HE COSTS OF AT TE NDI NG T RA IN I N G?


Yes 133 75.57%

No 43 24.43%

Documents

DEPARTMENT OF HEALTH AND AGEING …...Health Outcomes International The Department of Health and Ageing Evaluation of training and resources developed for the implementation of Stage