DESIGNING AND CREATING A RESIDENCY DIDACTIC CURRICULUM TO ENHANCE ABFM BOARD PASS RATE IN-TRAINING EXAMINATION (ITE) PEFORMANCE IN SMALL RURAL RESIDENCY PROGRAM

Mimo Rose Lemdja, M.D

UNIVERSITY OF ARKANSAS FOR MEDICAL SCIENCES REGIONAL CENTERS (MAGNOLIA) RURAL PROGRAM

Abstract

Background and Objective: The aim of this project is to design a residency didactic

curriculum to boost the Board pass rateITE performance in small rural Residency Programs. like

ours that are struggling to meet the ACGME requirement. The question this study seeks to

answer is if a residency curriculum that includes a robust mandatory series of weekly didactics,

online weekly independent assignments, and monthly Board review sessions can improve the

Board pass rateITE performance in small programs. The hypothesis for this project is that by

structuring targeted educational elements, including strengthening mandatory didactic

conferences, incorporating a well-designed online weekly curriculum, and installing monthly

Board review sessions, the Board pass rateITE performance in a small program can meet or

exceed the Board pass rate requirementnational mean for the exam. In general, a number of

studies examining the predictive power of ITEs success on the corresponding certification is

positive. The ITE exam is a useful predictor of initial ABFM certification examination

performance..

Method: Nine (9) residents from our small rural residency program were given a new

educational curriculum at the beginning of residency year in July 2019. The new curriculum was

similar to the previous one with three major changes: 1) Mmandatory weekly didactic attendance

(every Thursday from 12-5pm), 2) Oonline learning tool purchased by program called weekly

curriculum with Program Director assigning reading materials with weekly question bank, and 3)

Aa monthly Board review protected time where previous ITE exams are dissected and analyzed

in teams of 3 residents with faculty supervision. At completion, the resident ITE scoreBoard pass

rate will be compared to a control group. The control group in this study will be ITE score

ABFM pass rate from the past 23 years – 2017, and 2018.

Results: With the 2020 data not available as of this writing, this study will use the 2019

ITE as the baseline data. With the 2020 ABFM not available as of this writing, this study used

the 2019 ITE score as a preliminary tool of comparison. This project will only focus on ITE

performance improvement and the ITE performance in past years (2017 and 2018) will be used

as my control group. Out of nine residents, seven, or 77.8%, met requirement. TThhe baseline

data study group showed to be significantly higher ITE scores compared to the control group.

More than three- fourths of the study group ranged 80-94% in their In-Training examination,

which correlates with Board pass rate.

Discussion: The study group, which used an educational intervention consisting of mandatory

weekly didactics, online weekly assignments, and monthly board review session improved in

their in-training examination scores. This study suggests the improvement in exam scores is due

to the knowledge gained by protected time for reading and reviewing Board relevant materials.

Good performance in ITE correlates with ABFM exam passing. It is important to note that there

were limitations of only four months of curriculum prior to ITE and therefore could have

influenced the results.

Introduction

Residents in an ACGME- accredited Family Medicine program are eligible to take an

annual in-training examination. The aim of the ITE is to provide an assessment progress of the

residents in acquiring the knowledge that they need to be family physicians. It also provides a

residency program with a relative data about how the program as a whole is meeting the

educational goals (Winkel, 2010). The ITE is usually administered annually in an online format

during the last week of October.

The ITE consists of 200 multiple- choice questions and uses a content outline, which is

aligned to the design for the ABFM certification examination. ITE questions are written by

ABFM board- certified family physicians that may be in private practice or in academic

medicine. The committee reviews questions before they are published. The committee consists of

former or current residency program directors. After the completion of the ITE, one can

download the critiques and answers for all the exam questions in the physician portfolio. The

critiques give a brief rationale for the answers as well as including the recent reference, which

provides sources for further review.

The in-training examination is usually scored using a statistical analysis as those used for

the ABFM family medicine certification examination. The Since the objective of ITE ultimate

goal is to assess the residents’ , progress over their years of residency training., The the ITE

offersffers no passing score. but it has a mean score but tThe performance report provides

specific areas for resident’sone’s improvement and can be used to develop a resident’sn

individual strategy. .

According to the recent survey, family medicine programs offering online learning

curriculum had high pass rates on ABFM board exam. Lack of residency faculty qualified to

teach most subjects and competing curriculum priorities may contribute to most programs,

including our, who fail to meet the ACGME requirements (Gregg, 2008). Thus, majority of

residency programs do not offer a proper online curriculum. As a result, numerous family

medicine residents feel that they are not prepared enough to adequately counsel patients. To

enhanceing the ABFM Boards pass rate in small rural Residency Program that is struggling to

meet the ACGME requirement for initial Board certification, it is therefore important that a

mandatory weekly didactic, online weekly curriculum and monthly board review session’s

intervention to be implemented in rural residency to address this challenge. The online

curriculum content is highly interactive and covers most of areas, which may not be addressed in

medical education such as physical activity, nutrition, care of chronic medical conditions and

preventative medicine (Skye, 2011).

Online weekly curriculum with Program Director assigning reading materials with

weekly question bank and a monthly Board review didactic curriculum addresses numerous of

problems that residency programs experience struggling to attain the ACGME requirement.

Other scheduling issues include conflicts due to scheduled lecturers and duty hours (Batalden,

2013). Residents can access the materials at any time and place to address these challenges due

to the flexibility provided by online curriculum. Consequently a mandatory weekly didactic,

online weekly curriculum and monthly Board review sessions may prove more curricular and

cost- effective than traditional didactic-based education program alone.

Although many have hypothesized that web-based learning would result to in poor

performance and pass rate, in contrary, studies show that it improves performance (Kohlwes,

2006). This may be due to the elevated ability to standardize content and teaching plans. Many

studies show that online learning approaches in terms of competency results and residents’

knowledge offers advantages over traditional approaches and provides improvements in medical

knowledge. However, to date there are no programs of the duration and scope that include a

combination of mandatory weekly didactic, online weekly curriculum and monthly board review

sessions that have been implemented fully and evaluated thoroughly.

The aim of in-training examination is to assess the general medical knowledge, and to compare

the progress and scores from one year to another (Gregg, 2008). Different studies have shown

that the ITE is a reliable and valid estimate of resident knowledge and can be used to predict the

final performance. This can be a crucial tool to determine whether a mandatory weekly didactic,

online weekly curriculum and monthly Board review sessions learning approaches can enhance

the ITE performance and thereby ABFM pass rates in small rural residency.

Methods

This study aimed to use a mandatory weekly didactic, online weekly curriculum and

monthly Board review sessions learning approach and the ITE to measure the baseline level of

medical knowledge of residents and trends in improvement of pass rate.

We performed an analysis of ITE scores of nine (9) family medicine residents that

enrolled from year 2017 and through 2019 who were grouped in three teams each counseling

consisting of three residents.

Nine(9) residents from our small residency program were required to attend weekly

didactic at our family medical center where they received lectures, participated in clinical M&M

and presentations. Topics addressed during didactic by lecturers and residents ranged from adult

medicine, care of surgical patients, care of women, care of children and neonates, preventive

medicine, behavioral science, and mental health. Didactics were held at the FMC and included

both medical and pharmacy and audience (residents, medical and pharmacy students) were all

located at the FMC. The sessions occurred every Thursday afternoon. Residents evaluated the

lecture topic and the presentation using a 10-item evaluation form. Some residents found it very

time consuming and inconvenient when on away rotation or night float rotation. The resident

presentation topics were chosen by the PD and always reflecteds the subject where the resident

had a lower score on the previous ITE. To assess the effectiveness of the didactic, a conference

evaluation form is completed by residents every week. We also monitored conference attendance

rates by having an attendance sign-up sheet.

At the beginning of the residency year in 2019, the program purchased an online learning tool,

called Weekly Curriculum. This curriculum uses clinic cases and board-styled MCQs keyed to

essential elements from the ABFM-focused Family Medicine. The PD customizes our own

curriculum and advanced analytics with trainee and PD logbooks indicate strengths and

deficiencies, along with milestone reporting. The PD also customizes weekly emails reminders.

The curriculum offers thousands of questions before ITE and and also offers the ultimateserves

as a remediation resource when resident scores are back. A personalized mentor remediation

schedule will offer readings, teaching slides, and key MCQs to sharpen the resident areas of

identified weakness. When annual ITE results indicate weaknesses, PDs can now assign focused

Medicine readings and Board questions for individual residents. Reporting emails keep PDs

informed on progress, and tracking metrics keep the resident guided for improvement. The

online tool also provides residents and programs the perfecta means of measuring and sharpening

knowledge in medicine, with a singular focus on nailing success in the written Boards, ITE

Review exercises for each Board topic and ITE Exam Prep emulating the ABIM Qualifying

Exam. The control your test environment tool allows residents to take exercises and exam as a

group and learn as they go, or mimic the exam experience and take individually

Residents also have mandatory in-person monthly Board review sessions usually the last

Thursday of each month during the didactic protected time from 12-5pm. Residents are divided

in groups of 3 with group named Team 2016, Team 2017 and Team 2018. Each team has 20

questions from the assigned topic of the month and during the sessions, each group will have one

and half hours to conduct review sessions.

Results:

Three of the nine residents (33.3%) did not complete the mandatory weekly didactic from

July to October due to the residents being on Night Float or on an away rotation and were

exempt from participating in weekly didactic. Weekly curriculum assignments, had three

residents who did not complete their assignments, and only three completed theirs in a timely

manner. For the monthly board review sessions, only one team was consistently present every

month with a well elaborated review of assigned ITE questions in the first three months.

Chart1: ITE 3-Year Trend Chart

Table1.1 2

3 2017

Graph 1, 2, and 3: ITE Mean Scaled Score Graph 2019, 2018, and 2017 respectively

Discussion:

The study groups were required to complete at least 80% of the weekly didactic, 100% online

weekly curriculum assignments and monthly board review session. The ITE results from the past

two previous years were used as a control group and the 2019 ITE was the study group results.

The PGY-1 results stay about the same but looking at the PGY-2, there was a drastic increase in

ITE score in 2019 and this could be explained by the age and maturity of the resident and the

medical knowledge gained while in training but mostly by the 100% compliance in the online

weekly curriculum assignments.

Our program size offers a bigger challenge, the smaller the number of resident, the bigger the

variation. PGY-3 last year was composed of 3 residents and this year only 2 PGY-3 residents

took the ITE, also most PGY-1 and PGY-3 were on an out of town rotation between July and

October. The new curriculum showed some performance improvement in the ITE score for

PGY-2 but not for PGY-3, one of the reason could be the fact that PGY-3 did not finish their

online assignments like the PGY-2 did. The lack of significant difference in the PGY-3

performance could also be contributed to the fact that the PGY-3 did not complete their online

assignments as recommended. ITE score is a predictor of ABFM passing and there is no major

difference in program total score improvement when you compare to the last 2 years with this

year(2019) ITE score.

It is important to notice that there is no significant difference between 2018 and 2019 ITE score

when it comes to program total improvement even though PGY-2 residents had a significant

increase in their 2019 score. But the fact that the PGY-1 and 3 did not have the same increase in

their 2019 ITE score may be due to the fact that they did not comply with the new curriculum,

and did not complete all their assignments. This most likely contributed to the lack of significant

total program performance improvement.

Some residents not held accountable for the completion of their online assignments causes a

negative consequence on the evaluation of this new curriculum in terms of improvement of the

resident ITE score and anticipated increase in Board pass rate.

Moving forward, to best evaluate this new curriculum as an effective tool for improving the

Board pass rate by improving performance in ITE exams, all residents must be held accountable

for completing the online weekly curriculum assignments.

References

Leigh TM, Johnson TP, Pisacano NJ. Predictive validity of the American Board of Family

Practice In-Training Examination. Acad Med 1990;65(7):454-7.

Anandaraja, N., Hahn, S., Hennig, N., Murphy, R., & Ripp, J. (2008). The design and

implementation of a multidisciplinary global health residency track at the Mount Sinai

School of Medicine. Academic Medicine, 83(10), 924-928.

Batalden, M. K., Warm, E. J., & Logio, L. S. (2013). Beyond a curricular design of convenience:

replacing the noon conference with an academic half day in three internal medicine

residency programs. Academic Medicine, 88(5), 644-651.

Gregg, S. C., Eisenberg, D., Duffy, A. J., & Longo, W. E. (2008). Design, management, and

critical evaluation of a surgical basic/clinical science curriculum: the role of an

educational chief resident. Journal of surgical education, 65(1), 36-42.

Kohlwes, R. J., Shunk, R. L., Avins, A., Garber, J., Bent, S., & Shlipak, M. G. (2006). The

PRIME curriculum. Journal of general internal medicine, 21(5), 506-509.

Skye, E. P., Wimsatt, L. A., Master-Hunter, T. A., & Locke, A. B. (2011). Developing online

learning modules in a family medicine residency. Family Medicine-Kansas City, 43(3),

185.

Winkel, A. F., Hermann, N., Graham, M. J., & Ratan, R. B. (2010). No time to think: making

room for reflection in obstetrics and gynecology residency. Journal of graduate

education, 2(4), 610-615.

Geyman JP, Brown TC: A developing regional network residency program in family practice.

West J Med 121:514, 1974

Waxman H, Braunstein G, Dantzker D, et al. Performance on the internal medicine second-year

residency in-training examination predicts the outcome of the ABIM certifying examination. J

Gen Intern Med1994;9(12):692-4.

Lancaster CJ, Johnson AH, Hamadeh GN. Survey of family medicine residents evaluation

methods. Fam Med 1993;25(10):646-9.

Nathan RG, Mitnick C. Using an in-training examination to assess and promote the self-

evaluation skills of residents. (letter) Acad Med 1992;67(9):613.

The American Board of Family Medicine. Maintenance of certification. Part III— cognitive

expertise.

Available from: https://www.theabfm.org/moc/part3.aspx. Accessed December, 2019

Martin J, Meyer L. Updates from the Residency Review Committee for Family Medicine. The

Accreditation Council for Graduate Medical Education (ACGME). Available from:

http://www.acgme.org/

acgmeweb/Portals/0/PFAssets/Presentations/120_Family_Medicine_Review_Cycle_of_Cores_a

nd_subs.pdf. Accessed January 17, 2013.