INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS

INTER-RATER AGREEMENT IN KANSASSummer Principals Academy

July 22-24, 2014

Abilene, KS

2

What is Inter-Rater Agreement?• Consistent

understanding• Consistent

agreement• Consistent

process• Consistent

credibility

The degree 2 raters using the same scale give the same rating in identical situations.

3

Where is Inter-rater Agreement Useful??

Inter-Rater Agreement is useful in all areas of administrative practice including:

• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management

4

What are the Goals of Inter-rater Agreement?

• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.

• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility

5

How is Inter-rater Agreement Helpful??

• What is a Common Understanding?

• How does Inter-rater Agreement help administrators??

• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation

over Time • Passes Public Scrutiny

6

District-wide Agreement about External Criteria

• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols

7

Pragmatics/ConsiderationsPotential Landmines

• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the

Doubt• Central Tendency: Safety in

the Middle• Comparing One with

Another: Stick to the Rubric• Relying on Gut too Much:

Stick to the Rubric• Evaluating Absent

Evidence: Stick to What is Observed

8

How to Train Administrators in using Inter-rater Agreement

• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity

9

Getting Started in YOUR District

• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol

10

I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:

• Use a video from YouTube• Assess Specific Area of Teaching

• Student Engagement• Teacher Activity• Classroom Environment

2. Review current issue/goal/vision/needs

3. Review current practices/instrumentation• Rubrics• Protocol• Training

11

II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –

Ultimately to be used during Professional Learning Times with the Staffs

• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding

• “Demonstrates”• “Frequently”• “Usually”

12

II. Developing a Guidebook• Pieces and Parts (continued)

• Examples and Rubrics• Areas to Observe

• Establishes/maintains a culture of safety, respect and rapport

• Knows how to make content accessible to all learners

• Plans standards-based instruction and formative assessments

• Uses a variety of instructional strategies to meet needs

• Deepens understanding through critical thinking, problem-solving and student reflection

• Analyzes student performance and provides feedback

• Collaborates to support student learning

13

II. Developing a Guidebook• Pieces and Parts (continued)

• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of

agreement• Example Artifacts

• Classroom Rules• Lesson Plans• Parent Note Home

• Lesson Plan for Adults: Andragogy

14

III. Developing a Training Protocol• Step-by-Step Procedure

1. Initial Training: Goals/Terms• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples

2. Calibration of the Players• Video and Artifacts

3. Review of the System

4. Adjustments (If needed)

5. Implementation

6. Evaluate/Reflect/Adjust

7. Re-Calibrate…

15

• Start Slowly• Start with

“Engagement” with a Small Group

• Get Agreement• Gain Confidence

Over Time• Write and Refine in

Teams

Take Aways – To Do List

16

• Start with the End in Mind• Develop Common

Understandings• Use Groups to Ensure

Buy-In• Build Measurable Steps to

Reach Each Goal

Final Thoughts

17

Contact Information

Teacher Licensure and Accreditation Team DirectorScott Myers785-296-8010smyers@ksde.org