164

The Journal of Continuing Education in the Health Professions, Volume 20, pp. 164–170. Printed in the U.S.A. Copyright © 2000 The Alliancefor Continuing Medical Education, the Society for Academic Continuing Medical Education, and the Council on CME, Association forHospital Medical Education. All rights reserved.

Original Article

Evaluation of a Problem-Based Learning Workshop Using Pre- and Post-test Objective Structured Clinical

Examinations and Standardized PatientsPaul Davis, MB, ChB, FRCPC, FRCP(UK), Brent Kvern, MD, CCFP, Neil Donen, MB, ChB, FRCPC, Elaine Andrews, BA, APMR, and Olga Nixon

Abstract:

Background: Osteoporosis is a health care issue in which family physicians play a major role.Although awareness of osteoporosis is high, recent studies suggest that application of recentadvances in its treatment to the clinical setting may be low. We have developed a problem-basedlearning intervention for osteoporosis in which paired rheumatologists and family physiciansdeveloped nine problem-solving clinical scenarios. An educational matrix was used to link spe-cific case scenarios with individual teaching objectives, developed via a previous needs assessment.Family physicians participated in the workshop, developing best practice responses to the clini-cal scenarios with a trained facilitator and content expert.

Methods: To assess the impact of this intervention, family physicians participated in a pre- andpost-test evaluation, using objective structured clinical examinations and standardized patients.Objective structured clinical examination stations tested knowledge, skills, and judgment relat-ing to osteoporosis with respect to risk factors, use of appropriate investigations including bonemineral densitometry (BMD), strategies for the prevention of osteoporosis (both pharmacologicand nonpharmacologic), treatment options for established osteoporosis (bisphosphonates andhormone replacement therapy), and management of recent osteoporosis fracture. Participantswere evaluated using a predetermined score generated by their responses to objective structuredclinical examinations and standardized patients (max. score = 101). Evaluations were conductedanonymously, although participants had access to their own pre- and post-test results for per-sonal feedback. The impact of the workshop was assessed by comparing pre- and post-test responsesby group, by individual, and by station.

Results: Participants demonstrated a significant improvement in their post-workshop scores. Of40 participants, 26 showed improvement in score (>+10), 13 showed modest change (+1 to +10),and 1 showed a marked decrease (>–10). The greatest improvements were seen in the manage-ment of the male osteoporosis patient, determination of risk factors for osteoporosis, and the useand interpretation of bone mineral densitometry. Family physicians reported general satisfactionwith the content and format of both the workshop and the evaluation process.

Implications: We conclude that this type of problem-based learning intervention workshop resultsin improved knowledge, skills, and judgment in the management of osteoporosis by family physi-cians as objectively assessed using a pre- and post-test format including objective structuredclinical examinations and standardized patients.

Key Words: Osteoporosis, outcomes assessment, problem-based learning

Dr Davis and Ms Nixon: Continuing Medical Education,University of Alberta, Edmonton, AB; Dr Kvern: Facultyof Medicine and Dr Donen: Continuing MedicalEducation, University of Manitoba, Winnipeg, MB; and

Ms Andrews: Merck Frosst Canada Inc., Edmonton, AB.

Reprint requests: Paul Davis, MB, ChB, FRCPC,FRCP(UK), 562 Heritage Medical Research Centre,University of Alberta, Edmonton, AB T6G 2S2.

Today’s provider of continuing medical educa-tion (CME) must provide high-quality programswhose format and content both meets identifiedconsumer needs and incorporates the need tochange from a traditional to more behavioral CMEformat consistent with principles of adult learning.1

There is also a need to address the criticism, appro-priately leveled by some at CME providers, thatmuch current CME provided fails to result in prac-tice change or to influence patient outcomes.2–4 Thedevelopment of practice- or problem-based small-group learning interventions, in which physicianscan participate in interactive discussions of prac-tical and relevant case-based clinical issues, isone attempt to address this.5,6 To develop suchprograms requires an initial needs assessment andsubsequent development of attainable objectives.This format has been adopted by many CMEproviders and has been widely accepted by bothCME providers and consumers. Evaluation of theimpact of such CME programs is more difficult,and has frequently been criticized on method-ological grounds. Formal evaluations of outcomesfollowing CME interventions also can be timeconsuming and costly, and therefore cannot beused regularly.

Osteoporosis is one of this decade’s majorpublic health care concerns:7 it is associated withsignificant morbidity and mortality, resulting inmajor health care costs.8,9 A need to target osteo-porosis has arisen from an increased awareness ofthe socioeconomic impact of the disease, a recog-nition of the risk factors for its development, theability to objectively assess its extent and progressthrough bone densitrometry, and the developmentof therapeutic strategies for effective preventionand treatment.10–12 However, there is some evi-dence that these factors may not be translating intomeaningful practice change, both in Canada andthe United States.13,14 A recent needs assessmentby the authors found that both family physiciansand specialists have gaps in applying this infor-mation, including poor understanding of risk fac-tors for osteoporosis and variable use and inter-pretation of bone mineral densitrometry or of

appropriate prevention and treatment strategies.15

A needs assessment that included a review of theliterature, focus groups, and a self-administeredphysician questionnaire has recently been pub-lished.16 There are two reports concerning lowrecognition and treatment of osteoporosis amongin-patients admitted to tertiary care hospitals withosteoporotic fractures.17,18 In another study, at-risk seniors reported that they had good awarenessof osteoporosis problems, but that the informationthey received came rarely from their primary carephysician but from television or the lay press.19 Totry to address some of these issues, a CME inter-vention on osteoporosis for family physiciansusing the practice-based small-group learningintervention format was developed.20 The purposeof the current study was to make a formal evalu-ation of the impact and outcome of this program,using an objective structured clinical examina-tion and standardized patient pre- and post-testassessment of participants.

Methods

Educational Intervention

A single interdisciplinary focus group developeda series of learning objectives suitable for use inan educational intervention on osteoporosis, basedon a previous needs assessment.15 The learningobjectives were to acquire the ability to (1) describethe diagnosis and management of osteoporosis;(2) counsel patients on the risk factors and pre-vention of osteoporosis; (3) project the skills andjudgment required to more effectively use inves-tigations, especially bone mineral densitrometry;and (4) to appropriately review current practicebehavior, resulting in improved health care out-comes for patients. Three pairs of rheumatolo-gists and family physicians collaborated to ninecase scenarios that addressed the objectives. Aneducational matrix was developed, linking thecase scenarios with the predetermined objectivesbased upon identified needs. Participant and facil-

165

Davis et al.

itator manuals were developed as the principalresource for a 2.5-hour practice-based small-grouplearning intervention, which was facilitated by apreviously trained primary care physician: a sub-specialist was available as a content expert ifrequired. The program was accredited for 3.0MAINTENANCE OF PROFICIENCY M1 cred-its by the College of Family Physicians of Canada,Alberta Chapter. The program was initially eval-uated in a pilot project by 52 physicians, whogave it an overall mean satisfaction index of 3.40of a possible 4.00.20

Evaluation

Forty different primary care physicians participatedin the evaluation phase of this project, using apre- and post-test format in association with theeducational intervention described above. Partic-ipating physicians were recruited using a conve-nience sample and included physicians in bothrural and urban practice. To reduce bias, the edu-cation intervention was evaluated using the objec-tive structured clinical examination developed bythe Division of Continuing Medical Education atthe University of Manitoba.

Stations were developed to test knowledge,skills, and judgment relating to prevention andtreatment of osteoporosis and included two stan-dardized patients for assessing clinical and coun-seling skills. This evaluation process was devel-oped independently of the education intervention(workshop) and was used to assess content areasof the workshop. It also was designed to objectivelyassess changes in both physician behavior andknowledge. The pre- and post-tests were structuredidentically, with each physician acting as his or herown control. Station one consisted of a 49-year-old perimenopausal female patient inquiring abouther risks for osteoporosis and the value of hormonereplacement therapy. Station two presented a 68-year-old male with polymyalgia rheumatica receiv-ing therapeutic doses of oral corticosteroid. Par-ticipants were evaluated based on their ownresponses and on feedback by the standardized

patient at these two stations. Station three was acase scenario, in which participants answeredquestions relating to the risk factors for osteo-porosis and the appropriate investigation of themanagement of a 70-year-old female patient pre-senting to the emergency room with a vertebralcompression fracture. Station four involved inter-pretation of a bone mineral densitrometry reportin a 56-year-old perimenopausal female with arecent and past history of vertebral fracture: par-ticipants were requested to comment on appropriatetherapeutic interventions. This scenario was alsorepeated in a 68-year-old postmenopausal patient.At station five, appropriate advice was to be givento a menopausal patient on the risk-benefit ratioof hormone replacement therapy.

Assessments were based on duplicate, coded,written notes made by the participants during theencounter (i.e., chart review). The duplicate chartwas retained by the participant for subsequentpersonal feedback. Participants were evaluatedanonymously, according to a predetermined pro-tocol that assigned points (max. score = 101).There was no negative marking. Pre- and post-testresults were compared by group, by individual, andby station.

Results

Forty family physicians participating in the pre-and post-test evaluation phase of the study had par-ticipated in the osteoporosis problem-based learn-ing education intervention. Individual and groupresults were recorded using the predeterminedpoints system. For stations involving standard-ized patients, points were given based on boththe patient’s assessment of the physician and thephysician’s assessment of the patient. Mean pre-test scores improved from 65% (range 44–80) to78% (range 55–93) post-workshop. This changewas statistically significant (p = .001) by the pairedt-test analysis. Thirty-nine of the 40 (98%) par-ticipants improved from their pretest scores (range1–29). Twenty-six of these (65%) improved their

166

Evaluation of Problem-Based Learning Workshop

scores by 10% or more (Fig. 1). Mean improve-ment in individual objective structured clinicalexamination stations ranged from 8% to 18%(Table 1). The greatest improvements occurredin stations that evaluated the participants’ knowl-edge of risk factors for osteoporosis and appro-priate use and interpretation of bone densitrome-try. At both stations with a standardized patientstation there was a significant score increase (14%)for individual physicians’ communication skillsassessed by the standardized patients. There wasalso a marked improvement in the group score forparticipants’management of the male patient at riskfor steroid-induced osteoporosis; one physician’sscore fell on the post-test evaluation, however.Although the anonymity of the process made itimpossible to definitively explain this apparentaberration, one particular physician did express avery negative attitude about the evaluation processand may have been responsible for this result.Overall, the satisfaction index demonstrated thatthe evaluation process was well received by the

majority of participants. In post-program evalua-tions, 33 of 38 (87%) participants said they wouldparticipate in this type of learning format andevaluation again, and 30 of 37 (81%) said theywould recommend this format to a colleague.These results were similar to those observed in pre-vious pilot studies of the workshop.20

Discussion

Based on a previously conducted community physi-cian needs assessment, a small-group problem-based learning program on osteoporosis for fam-ily physicians was developed.15,20 This type ofeducational intervention has proven to be popularwith consumers and providers alike, and has beenshown to be an effective education activity by sat-isfaction indices in post-workshop evaluations.Formal evaluation of the value of such programsassessing physician knowledge, skills, and judg-ment and the impact on patient outcomes is rare.In the present study, we sought to formally assess

167

Davis et al.

Figure 1 Change in individual physician pre- and post-workshop score.

the impact of our program, using an objectivestructured clinical examination evaluation thatincorporated standardized patients. Although objec-tive structured clinical examinations and stan-dardized patients have been used in various edu-cational settings, their use in CME programminghas been reported infrequently, and usually in rela-tion to needs assessments or construction ofimproved educational tools for other physicians.21–24

This issue has recently been reviewed.25 Nonethe-less, this evaluation format has been shown to beboth valid and reliable when the content of theobjective structured clinical examination directlyrelates to the objectives of the educational inter-vention.26 We therefore chose objective structuredclinical examinations with standardized patients toevaluate our program, since we felt it would assessboth the participants’competence and performance.

We are confident that our previously describedneeds assessment on osteoporosis15 allowed us todevelop an educational intervention that was rel-evant and applicable to primary care physicians.Passive educational interventions such as didacticlectures require only superficial processing of

information, and this type of learning style isunlikely to challenge or change long-held attitudesand assumptions about treatment. Active and inter-active educational strategies are more effective ingenerating deeper cognitive processing that leadsto both attitudinal and behavioral changes.1,2,4,27

In many areas of medical education, there is dis-agreement on what constitutes clinical compe-tence, let alone how to measure it.28 Also, trainingprograms for primary care physicians often lackmethods for valid and reliable performance assess-ment.29 An objective evaluation method such asusing anonymous pre- and post-test objective struc-tured clinical examinations in which each physi-cian acts as his or her own control seemed morevalid and reliable. Certainly, other CME programshave found that community physicians are reluc-tant to be involved if the evaluation is perceived notto be confidential.21 By incorporating the stan-dardized patient component into the objectivestructured clinical examination evaluation, bothphysician competence and performance could beassessed. In the pretest, group and individual deficitscould be identified against an objectively defined

168

Evaluation of Problem-Based Learning Workshop

Table 1 Mean Pre- and Post-Test Scores by Station

Station Description Pretest Post-TestResults (%) Results (%)

1 Forty-nine-year-old female asking about an article she read about menopause

a Actor’s assessment of physician 69 83b Participant’s assessment of patient 74 82

2 Sixty-eight-year-old male with polymyalgia rheumatica on long-term steroids with back pain

a Actor’s assessment of physician 64 78b Participant’s assessment of patient 61 77

3 Thinking about the diagnosis: important risk factors and when to order a bone mineral densitometry 59 77

4 Established osteoporosis*: interpret a bone mineral densitometry report and manage accordingly†; manage a patient in the ER with acute vertebral compression fractures 68 77

5 HRT‡ = specific risk and benefits in selected clinical situations 62 70TOTAL 65 78

*Bennet and Casebeer, 19951; †Harris et al., 199311; ‡HRT = hormone replacement therapy.

clinical competence check list, which validatedour previous needs assessment.

Some workers have questioned whetherassessment of competence in this fashion is pre-dictive of physician performance.30,31 However,objective structured clinical examination evalu-ations and standardized patients are widely used,since others accept that clinically competent per-formance is multifaceted and that these evaluationsare therefore valuable.4 Based upon the results offormal pre- and post-test evaluations of individ-ual physicians, we feel it likely that the observedimprovements in test scores reflected an increasenot only in their knowledge but in their skillsand judgment in dealing with objectively obtainedneeds. This conclusion was also supported bythe informal feedback from participants, whoindicated that they expected to perform at a moreclinically competent level as a result of a behav-ioral change learned through participation in theeducational intervention. However, more researchin this area of CME is required to determinewhether such evaluations of competence do in factlead to significant behavioral change in perfor-mance.

Although we are encouraged by the results ofour evaluation of our problem-based learning pro-gram, some limitations need to be acknowledged.First, our physician group was small and a con-venience sample. We were unable to incorporatea control group. Finally, the pre- and post-work-shop objective structured clinical examinationevaluation may have enhanced learning duringthe workshop intervention. Certainly, the post-workshop evaluation reinforces knowledge byrequiring its application and recall shortly after theworkshop (the Hawthorne effect).

In conclusion, we have developed a small-group problem-based learning workshop on osteo-porosis for family physicians. Using an objectivestructured clinical examination format incorpo-rating standardized patients, we conducted a pre-and post-test evaluation of the workshop. Thepretest also served as an additional needs assess-ment on osteoporosis for our participants, and val-

idated the objectives of the workshop developedfrom our previous needs assessment. The impactof our program was evaluated using a similar post-test format, which demonstrated significant changesin knowledge, skills, and attitudes and improvedcompetence based on the individual physician-standardized patient interaction. We are hopeful thatin the near future we will conduct a formal assess-ment of physician performance through chart auditof those who participated in our program.

References

1. Bennett N, Casebeer L. Evolution of planningin CME. J Cont Educ Health Prof 1995;15:70–79.

2. Davis D, Thomson M, Oxman A, Haynes B.Evidence of the effectiveness of CME. JAMA1992; 268:1111–1117.

3. Glazier R, Buchbinder R, Bell M. Criticalappraisal of continuing medical education inthe rheumatic diseases for primary care physi-cians. Arthritis Rheum 1995; 38:533–538.

4. Davis D, Thomson M, Oxmon A, Haynes B.Changing physician performance. JAMA1995; 274:700–705.

5. Premi J, Shannon S, Hartwick K, et al.Practice based small group CME. Acad Med1994; 69:800–802.

6. Premi J. Problem-based, self-directed continu-ing medical education in a group of practicingfamily physicians. J Med Educ 1988;63:484–486.

7. Raisz LG. The osteoporosis revolution. AnnIntern Med 1997; 126:458–462.

8. Norris RD. Medical costs of osteoporosis.Bone 1992; 13:S11–S16.

9. Chrischilles E, Shireman T, Wallace R. Costsand health effects of osteoporotic fracture.Bone 1994; 15:377–386.

10. Black D, Cummings S, Karpf D, Cauley J,and the Fracture Intervention Trial ResearchGroup. Randomized trial of effect of ale-dronate on risk of fracture in women withexisting vertebral fracture. Lancet 1996;348:1535–1541.

169

Davis et al.

170

Evaluation of Problem-Based Learning Workshop

11. Harris S, Watts N, Jackson R, et al. Four yearstudy of intermittent cyclic etidronate treat-ment of postmenopausal osteoporosis. Am JMed 1993; 95:557–567.

12. Adachi J, Bensen W, Brown J, et al.Intermittent etidronate therapy to prevent cor-ticosteroid-induced osteoporosis. N Engl JMed 1997; 337:382–387.

13. Scientific Advisory Board, OsteoporosisSociety of Canada. Clinical practice guidelinesfor the diagnosis and management of osteo-porosis. Can Med Assoc J 1996;155:1113–1133.

14. American College of Rheumatology TaskForce on Osteoporosis Guidelines.Recommendations for the prevention andtreatment of glucocorticord-induced osteo-porosis. Arthritis Rheum 1996; 39:1791–1801.

15. Suarez-Almazor M, Messina D, Homik J,Davis P. Attitudes and beliefs of family physi-cians and gynecologists in relation to the pre-vention and treatment of osteoporosis. J BoneMiner Res 1997; 12:1100–1107.

16. Lockyer J, Hanley D, Fidler H, Toews J,Lysholm-Andrews E. Physician educationalneeds in osteoporosis: an approach to needsassessment. J Cont Educ Health Prof 1998;18:185–189.

17. Hajesar E, Hawker G, Bogoch E. Follow-uptreatment for osteoporosis in a fragility frac-ture population: an assessment of practice pat-terns in the fracture clinic. J Bone Joint Surg1999 (in press).

18. Juby A, DeGeus C, Davis P. Evaluation of riskfactors for osteoporosis, its diagnosis andtreatment in seniors admitted with hip fracture[abstract]. J Rheumatol 1999; 26:1632.

19. Juby A, McAuley L. Assessment of osteoporo-sis awareness and treatment in Albertanseniors [abstract]. J Rheumatol Suppl 1998;52:57.

20. Davis P, Andrews E, Donen N, Juby A, KvernB, Low S, et al. Case studies in osteoporosis, aproblem-based learning intervention for fami-ly physicians. J Rheumatol 1999;26:2418–2422.

21. O’Brian M, Feldman D, Alban T, et al. Aninnovative CME program in cardiology for

primary care practitioners. Acad Med 1996;71:894–897.

22. Sloan D, Donnelly M, Schwartz R, et al. Themultidisciplinary structured clinical instructionmodule as a vehicle for cancer education. AmJ Surg 1996; 172:220–225.

23. Lockyer J, El-Guebaly N, Simpson E, et al.Standardized patients as a measure of changein the ability of family physicians to detectand manage alcohol abuse. Acad Med 1996;71:S1–S3.

24. Davis P, Russell A, Skeith K. The use of stan-dardized patients in the performance of aneeds assessment and development of a CMEintervention in rheumatology for primary carephysicians. J Rheumatol 1997; 24:1195–1199.

25. Rethans J. Needs assessment in continuingmedical education through standardizedpatients. J Cont Educ Health Prof 1998;18:172–178.

26. van der Vleuten C, Swanson D. Assessment ofclinical skills with standardized patients: stateof the art. Teach Learn Med 1990; 2:58–76.

27. Onion C, Bartzokas C. Changing attitudes toinfection management in primary care: a con-trolled trial of active versus passive guidelineimplementation strategies. Fam Pract 1998;15:99–104.

28. Capos-Outcall D, Watkins A, Fulginiti J, et al.Correlations of family medicine clerkshipevaluations and objective structured clinicalexamination scores and residency director rat-ings. Fam Med 1999; 31:90–94.

29. Wasyluk J, Wegrzyn Z. Problem of objectiveperformance assessment of primary healthcare physicians. Scand J Prim Health Care1991; 9:276–280.

30. Ram P, van der Vleuten C, Rethans J, et al.Assessment of practicing family physicians:comparison of observation in a multiple sta-tion examination using standardized patientswith observation of consultations in dailypractice. Acad Med 1999; 74:62–69.

31. Rethans J, Sturmans F, Drop R, et al. Doescompetence of general practitioners predicttheir performance? Comparison betweenexamination setting and actual practice. BMJ1991; 303:1377–1380.