RESIDENCY-IN-TRAINING EVALUATION (RITE) 2013

Lenora Fernandez, MD, FPCP

OVERVIEW on RITE

• RITE/REE held yearly 1987-2003 then stopped when validity & reliability in doubt

• Voluntary on-line format 2009-2011: very few takers

• Annual written exam format resumed 2013 = OCTOBER 13, 2013, Sunday

OBJECTIVES of RITEGeneral Objective:• To determine the cognitive skills of internal medicine residents in all

PCP-accredited training programs.

Specific Objectives:– To compare the cognitive skills of IM residents among all accredited

internal medicine training programs in the Philippines.– To compare the cognitive skills between 2nd and 3rd year IM residents

within the institution and all other institutions.– To determine the areas of strengths and weaknesses of the examinees

and serve as guide in maintaining and improving their training programs.

– To provide a tool in setting standards for accreditation of IM training programs to the Committee on Accreditation in (future)

– To determine if this examination can predict the performance of the residents in the PSBIM.

OVERVIEW on RITE• Multidisciplinary committee

• Tried to simulate the PSBIM• 200 items: 175 MCQs, 25 Modified True or False

• Chapters hosted and conducted the RITE

CardiologyPulmonary MedicineEndocrinologyGastroenterologyNephrologyInfectious DiseasesHematology

Allergology-ImmunologyRheumatologyMedical OncologyNeurologyDermatologyEthics

Recall - 30% Based on PCP Glossary of TopComprehension – 30% DiseasesAnalysis – 40%

23 RITE TEST SITESFatima College of Medicine, Valenzuela City (CAMANAVA)Laoag City Dagupan City Baguio City Tuguegarao CityLegaspi CityLipa City Tacloban City Davao City Cagayan de Oro City Zamboanga City

Quezon CityManila City Marikina City Makati City (MATAPAT)Las Pinas City (PAMUNLAS)Pasay CityPasig City (PASJMAN) Angeles City, Pampanga Cebu CityTagbilaran City Iloilo CityBacolod City

RITE DEMOGRAPHICSTotal number of examinees 1264, 94 training institutionsYear Level 2 examinees N = 476

Mean score = 102.38/200Year Level 3 and 4 examinees (no.) 469

Mean score = 106.1/200Post-residency examinees (no.) 319

Mean score = 94.7Set minimum passing level*:

Year Level 2Year Level 3, 4 and post-residency level

86 out of 200107 out of 200

RITE exam analysis:Range: Mean: Median:Mode: Reliability index:

65.2-153.8101.8100.8108.20.83 (vs 0.63-0.78 in previous)

* MPL pre-determined: YL 2 – 10% lower than YL3/4 (71% of 200 items); MPL lowered by 1 SD after results collated

RITE TEST INDICES

Difficulty index:

10% very easy25% easy15% moderate30% difficult20% very difficult

< 1% very poor50% poor30% good20% excellent

Discrimination Index:

MEAN RITE SCORES IN 3 GROUPS

80

85

90

95

100

105

110

2nd year 3-4th year Postgraduate

102.4 + 14.6106.7 + 15.3

94.7 + 11.67

N = 476 469 319

Mea

n RI

TE S

core

s(o

ver 2

00 it

ems)

p < 0.001

p < 0.001

PERFORMANCE IN DIFFERENT TOPICS OF THE 3 GROUPS

0.00%

20.00%

40.00%

60.00%

80.00%

YL2 YL3-4 PGRAD% mean score

in topic

TOP 25 TRAINING INSTITUTIONS IN PERCENTILE RANK, RITEUP-PGHNational Kidney & Transplant InstituteUniversity of Santo TomasPremiere Medical CenterCardinal Santos Medical CenterZamboanga City Medical CenterOspital ng Maynila Medical CenterFEU-NRMF Medical CenterMakati Medical CenterDavao Doctors HospitalWestern Mindanao Medical CenterSt Lukes Medical CenterUERM Memorial Medical CenterSLU-Hospital of the Sacred HeartChong Hua HospitalWest Visayas State University Medical CenterUniversity of Perpetual Help Dr. Jose G. Tamayo Medical Center-Binan, LagunaSouthern Philippines Medical CenterDivine Word HospitalChinese General Hospital and Medical CenterSt. Paul's HospitalAFP Medical CenterManila Adventist Medical CenterMary Mediatrix Medical CenterDr. Pablo O. Torre Memorial Hospital

YEAR LEVEL 3 RITE - PSBIM

0

20

40

60

80

100

120

Passed PSBIM Failed PSBIM Did not take PSBIM

94.45 + 9.38114.41 + 12.84

98.79 + 13.64

p < 0.001

Mea

n RI

TE S

core

s(o

ver 2

00 it

ems)

N = 232 51 186

POSTGRADUATE EXAMINEES:RITE - PSBIM

8486889092949698

100102104

Passed PSBIM Failed PSBIM Did not takePSBIM

93.5 + 10.16

102.7 + 10.4

90.75 + 10.4

p < 0.001

Mea

n RI

TE S

core

s(o

ver 2

00 it

ems)

N = 79 116 124

FEEDBACK OF EXAMINERS/PROCTORS• Clear instructions? YES- 100%• Enough time to prepare for organization of RITE? YES – 96.1% • Conflicts in the instructions given? NONE 78.7%• Do you agree to the involvement of PCP chapters in the conduct of

RITE? YES – 99%• Will you be willing to serve again as proctor or organizer for the

RITE in the coming years? YES- 98%• Further comments:

– Role of RITE in regular evaluation of residents?– Notify earlier, orientation meeting, minimize conflicts in instructions– Separate exam for 2nd and 3rd years– Include first years– Chairmen to encourage post-residency members who want to take PSBIM– Compensation to staff?

FEEDBACK OF EXAMINEES• Was RITE given at the right time of your training? YES 86.7%• For residents-in-training, test questions relevant and appropriate to your

level of training? YES 98.3%• For post-graduate examinees, test questions serve as a “good review” for

your coming PSBIM exam? YES 98.4% • Enough time allotted to answer all questions? YES 96.2% • Enough time to review for the exam? YES 55.33%• Details provided beforehand sufficient? YES 87.6%• Venue conducive to answering exam well? YES 91.2%• Proctors helpful in clarifying your questions during the exam? YES 99.7%• Will you recommend that RITE be regularly conducted on an annual basis?

YES 95.1%• Would you prefer RITE to be conducted on-line? YES 37.4%

FEEDBACK OF EXAMINEES

• Must include first years, more often• Venue issues (temp, announcement, chairs/desk,

comfort room, crowding)• Questions (different from PSBIM, relevance)• No questions allowed• Coverage/blueprint not announced earlier• Must be paired with a post-exam review

CONCLUSIONS ON RITE ANNUAL WRITTEN EXAM:• The RITE 2013 met most of specific objectives it aimed for:

– compare the cognitive skills of IM residents among all IM training programs. – DONE

– compare cognitive skills between 2nd and 3rd year IM residents within the institution and all other institutions. - DONE

– determine the areas of strengths and weaknesses of the examinees and serve as guide in maintaining and improving their training programs. – DONE

– determine if this examination can predict the performance of the residents in the PSBIM. Based on the results, it seems it can predict the performance in the PSBIM.

• Reliability index high at 82.9% + opinion of majority among the examiners and examinees to continue with RITE.

• The RITE is recommended to be continued.

RECOMMENDATIONS• Reformulation of RITE Committee examiner composition . • More training in test construction.• Next date: October 12, 2014, Sunday. • More test sites nearer to examinees to be manned by Chapters.• Better and earlier coordination with the PCP chapters.• Test blueprint given today:

– ALL MCQ’s– 250 items – include questions on research and principles on critical

appraisal of literature – Integrative questions across different topics;subdiv to ambulatory,

emergency, or critical care dimensions.• Scantron machine already available at PCP• More efficient system of dissemination and recording/collating • Data to be continually analyzed • Discussion on bearing on internal evaluation of resident will be

opened?

Thank you!Dr. Lenora Fernandez ChairDr. Adrian Rabe Assistant Chair

Dr. RontgeneSolante IDSDr. Sonia Salamat IDSDr. Beatriz Concepcion NephrologyDr. Elaine Cunanan EndocrinologyDr. Heizel Reyes Rheumatology/EthicsDr. Norman Maghuyop Pulmonary MedicineDr. Jesus Relos HematologyDr. Dennis Sacdalan OncologyDr. Richard Tiongco CardiologyDr. Eternity Labio GastroenterologyDr. Cherie Cervantes Immunology-Allergology

Adviser: Regent Nenita A. Collantes, MD, FPCP