Inter-Rater Reliability RespiratoryIvy Tech Community College-Indianapolis

What Is Inter-Rater Reliability (IRR)?• The extent to which two or more raters agree• A fair measurement of student competency • Addresses the uniformity of the implementation of evaluation

systems being utilized

Importance of IRR• Required by CoARC • Accreditation Standard Requirement:

• Student Evaluation 3.09, 3.10, 3.11• CoARC Standard 3:09: The program must conduct and document

evaluations with sufficient frequency to keep students apprised of the progress toward achieving the curriculum competencies, and to allow immediate identification of learning deficiencies and the development of a means for their remediation in a reasonable time frame.

• CoARC Standard 3:10: The program must administer evaluations uniformly and equitably to all students in the program for didactic, laboratory, and clinical education components

• CoARC Standard 3.11: The program must develop processes that facilitate the development of inter-rater reliability among those individuals who perform student clinical evaluations

Importance of IRR• IRR requires some level of objectivity• IRR provides validation of evaluation results• Certify skills competency• Assign the appropriate “grade”

• Increases “confidence” that people are following the guidelines in a similar method

Ensuring IRR—per CoARC• Respiratory Programs should:• Implement Preceptor training on a regular and repeating basis• Training should include:

• Roles and responsibilities of preceptors• Direct observation skills• Effective feedback and evaluations• Inter-rater reliability exercises (e.g. evaluation of video

demonstration)• Keep records

Goal of Presentation• Goal of presentation• Provide insight into the student evaluation process

• How to perform objective evaluations• Achieve consensus amongst raters

• Describe the IRR process system and how it is used in the Ivy Tech Community College Respiratory Care Program-Indianapolis

Competency Domains• CoARC, our accreditation agency states that students must

demonstrate competency in 3 domains:• Knowledge• Technical performance• Behavior

• From an educational perspective this can present a dilemma!

The Dilemma• Evaluating student clinical performance and behavior is

difficult and can be somewhat subjective

• How do we remove the subjective?

• How can we get an objective record of the most meaningful behaviors?

Reliable Techniques• Most Educators would agree that the most valuable and

widely used techniques for clinical evaluation are:

• Anecdotal notes• Rating scales/procedural evaluations• Checklist/clinical objectives

Performance Evaluations• Clinical Performance Evaluations of Learning and

Development provide a means to DIRECTLY evaluate:

• Clinical competency• Technical performance• Behaviors

Validity/Procedure Evaluations• How can this be accomplished in a valid way?• Clinical paperwork

• Frequent and timely student evaluations including student feedback• Competency check-offs

• Procedure Evaluations or Competency Check-Off Forms:• Tool to evaluate clinical competencies • Have clear cut steps on what is being assessed• Has a rating scale

Example• A student who is performing unsatisfactorily is rated as

satisfactory creates the following problems:• Student thinks their performance is ok and will continue with

poor technique or behavior• Students aren’t provided objective feedback, and therefore

aren’t aware of their deficiencies of the need to make changes.• The ITCC faculty aren’t aware of the defiance's and necessary

corrective action can’t be provided

• ULTIMATELY we have failed the student and profession!

Completing the IRR Exercise• You will be viewing four procedures via video• The videos and competency check offs may be completed in one

time frame or if needed, individually • While watching the video complete the competency check off

form used in the Indianapolis region• Make sure to indicate your name, date, and facility at the top of

each competency check off form to document completion for tracking purposes

• Click on the submit button when competency check-off form is completed.

•All four competencies are due based on your clinical affiliates deadline, 2014!

Thank You• Thank you once again for your support in providing quality

respiratory care education• Please don’t hesitate to call the Director of Clinical Education

(DCE) of the Ivy Tech Community College-Indianapolis Campus at 317-921-4696 with questions!