An Introduction to Understanding the OT and OTA

Fieldwork Performance Evaluations

(FWPEs)

Karen Atler, MS, OTR Colorado State University, Fort Collins, CO

Roberta Wimmer, OTR/L Pacific University, Forest Grove, OR

2003 The American Occupational Therapy Association, Inc. www.AOTA.org

2

Task Force Members Carole Dennis, PhD, OTR

Ithaca College, New York Carole Hays, MA, OTR

Springfield Hospital Center, Maryland Becky Robler, MEd, OTR

Pueblo Community College, Colorado Karen Atler, MS, OTR, Co-Chairperson

Colorado State University Roberta Wimmer, OTR, Co-Chairperson

Pacific University, Oregon

3

Objectives

Describe the Entry-level practice competencies for

OT and OTA students Purpose, format, content, and scoring

of the companion evaluation forms New concepts and terminology used in

the FWPE from the OT Practice Framework

Begin to score items on the FWPE

4

Task Force’s Charge Revise/develop evaluation tools to measure

assistant and professional Level II fieldwork student performance.

Expectations Conduct review of literature across disciplines Synthesize feedback on current AOTA FWE/OT

forms Incorporate 1997 NBCOT Practice Analysis results Address identified desired characteristics

5

Desired Characteristics

Companion documents for assistant and professional level that Measure entry-level competence Focus on occupation-based practice Reflect current and future practice Can be used in a variety of settings Provide feedback to students Can be easily used in a timely manner

6

NBCOT Practice Analysis 1997

What OTs & OTAs Do

Determining needs/priorities for interventions

Identifying/designing interventions

Implementing interventions Reporting/evaluating

intervention effectiveness Providing OT services for

populations Managing delivery of OT

services Advancing effectiveness of

the OT profession

What OTs & OTAs Need To Know

Human development and performance

Principles/strategies in the identification/evaluation of strengths and needs

Principles/strategies in intervention/treatment planning

Principles/strategies inintervention

Nature of occupation and occupational performance

Service management Responsibilities as a professional

7

Standards of Practice for Occupational Therapy Identifies minimum standards Identifies key performance areas for the

OT and OTA Professional standing and responsibility Referral Screening Evaluation Intervention plan Intervention Transition services Discontinuation

8

ACOTE: Minimum Standards and Outcomes

OT Be a generalist Achieve entry-level

competence Articulate, apply, and justify

occupation interventions Supervise and collaborate

with the OTA Keep current with best

practice Uphold the ethics, values,

and attitudes of the profession

Be an effective consumer of research and knowledge

OTA Be a generalist Achieve entry-level

competence Work under the

supervision of and in cooperation with the OT

Articulate, apply, and justify interventions related to occupation

Keep current with best practice

Uphold the ethics, values, and attitudes of the profession

9

Goal of Level II Fieldwork Education for the OT and OTA Student

Develop competent, entry-level generalists

Include an in-depth experience in delivering occupational therapy services

Be designed to promote reasoning, enable ethical practice, and develop professionalism

10

The Process

Began with OTA evaluation Reviewed by experienced panel Submitted to COE Made revisions Completed pilot studies (2 OTA, 1

OT)

11

Design and Analysis of Pilot Studies: The Rasch Measurement Model

less able Student Ability more able

12

Results of Pilot Studies Good representation in pilot samples Students and educators preferred new form Good scale and response validity Inaccurate use of scale

Rating Scale Usage in Pilot Studies II and III

4 = Exceeds Standards

3 = Meets Standards

2 = Needs Improvement

1 = Unsatisfactory

Rating Scale Descriptors

2934

5356

1310

00

% Usage OTA% Usage OT

13

The FWPEs for OT and OTA Students Companion documents Terminology Content layout Purpose Design Rating scale Scoring system

14

FWPEs OT and OTACompanion Documents Focus

The occupational therapy process The clinical reasoning process Roles and responsibilities of the OT and OTA

Structure Collaborative process—student and FW

educator Same layout Same rating/scoring system

15

Terminology of the FWPEs Reflects

Standards of Practice for Occupational Therapy and ACOTE Education Standards

Occupational Therapy Practice Framework The glossary

16

Content Layout of FWPEs Summary Sheet Overview/instructions Organization of items Space for comments—midterm and

final Performance Rating Summary

Sheet

17

Content of OT and OTA Evaluations

OTA Fundamentals of practice (3) Basic tenets (3) Evaluation/screening (5) Intervention (6)

Communication (2) Professional behavior (6)

OT Fundamentals of practice (3) Basic tenets (4) Evaluation/screening (10) Intervention (9) Management of OT services

(5) Communication (4) Professional behavior (7)

18

Primary Purpose of the FWPEs

Measures entry-level competence Designed to differentiate the

competent student from the incompetent student

Not designed to differentiate levels above entry-level competence

19

Purpose (continued)

Provides student with accurate assessment of his or her competence for entry-level practice over time Growth occurs over time Midterm and final scores reflect this

change Midterm scores: Satisfactory–

unsatisfactory Final scores: Pass–no pass

20

Purpose (continued)

Provides feedback to student Provides opportunity for student

self-assessment

21

Design of the FWPEs

The “doing” of the occupational therapy process is evaluated, not the individual tasks in isolation

NOT all items are equal in level of difficulty (i.e., simple to complex)

Evaluation is supplemented with development of site-specific objectives

22

RASCH Ordering of Items OT

 

- -2- - - - - - - - - -

1- - - - - - - - - -

0- - - - - - - - - -

-1- - - - - - - - - -

-2-

Interpersonal; 41

Res

pnds

2 f

dbk

38D

iver

sity

; 42

Eth

ics;

1  C

olla

b c

supe

r;36

Wor

kbeh

39,

saf

ety

2S

teps

3,R

esp;

37

        Tim

emng

40; L

egib

le 3

4

  Cli

ent c

ente

red

inte

rv 2

2    O

ccup

n ba

sed

inte

r 23

  Sel

ects

rel

occ

u; 2

1P

rodu

ces

wor

k; 3

1C

lear

doc

umen

tati

on; 3

3O

rg g

oals

30;

ver

bl 3

2 D

ocum

ents

intr

v; 2

6C

olla

bora

tes

clnt

; 7L

angu

age

refl

ects

35

Doc

umen

ts e

val r

slts

17

OT

beli

efs

4 O

btai

n in

f 12

R

ole

of O

T; 6

Rat

iona

le T

x 18

, A

dmin

iste

r as

ses

13

Art

ic r

atin

al e

val;

8A

rtic

val

ue o

cc 5

Mod

ifie

s ap

proa

ch 2

4U

nder

st f

inan

ces;

29

Col

labo

rate

s O

TA

28

Occ

pro

f 10

, Adj

st a

ss 1

4E

st P

lan

16, A

sses

s ft

r 11

Upd

ates

; 25,

Evi

denc

e 19

In

terp

rets

eva

l res

lts;

15

  Sel

ects

rel

ass

es m

ethd

; 9  A

ssig

ns r

esp

OT

A; 2

7

HARD

 

23

Rasch Ordering of Items OTA

Easier

2- - - - - - - - - -

1- - - - - - - - - -

0- - - - - - - - - -

-1- - - - - - - - - -

-2-

Cultural competence; 25

Ethics; 1

Interpersonal skills;24

Responds to Feedback;21

Safety; 2 and 3

Work Behaviors; 22

Therapeutic Use Self;16

Written Communication;19

Verbal Communication; 18

Self-responsibility; 20

IImplements intervntn; 14

OT/OTA Roles 5DataGather

Selects Intervention;13

Activity Analysis; 15

Evidence BasedPractic 6

Plans Intervention; 12

Reports; 10

OT Philosophy; 4

Administer Assessmnts;8

Establishes Goals; 11

Modifies Intrven Plan;17

Interprets Assessment;9

Harder

24

Rating Scale of FWPEs

4 = ExceedsStandards

3 = MeetsStandards

2 = NeedsImprovement

1 = Unsatisfactory

Performance is highly skilled and self-initiated. This rating is rarely given and would represent the top 5% of all the students you have supervised.Performance is consistent with entry-level practice. This rating is infrequently given at midterm and is a strong rating at final.

Performance is progressing but still needs improvement for entry-level practice. This is a realistic rating of performance at midterm and some ratings of 2 may be reasonable at the final.

Performance is below standards and requires development for entry-level practice. This rating is given when there is concern about performance.

25

Scoring System of FWPEs Each item must be scored Ethics and safety items must be

passed Each item rating recorded on

Performance Rating Summary Sheet All items summed up at midterm and

final Score compared to scales provided

26

Midterm and Final Scoresof the FWPEs

OTAOverall Midterm ScoreSatisfactory: 54 & aboveUnsatisfactory: 53 &

belowOverall Final ScorePass: 70 & aboveNo Pass: 69 & below

OT Overall Midterm ScoreSatisfactory: 90 &

aboveUnsatisfactory: 89 &

belowOverall Final ScorePass: 122 & aboveNo Pass: 121 & below

27

Rating Performance Using the FWPEs

Case ScenariosJohn—OT Evaluation Mary—OT Intervention Sandra—OTA Evaluation/Screening David—OTA Intervention

28

Individualizing the FWPEs

Designed for additional objectives to be written to add clarification Site-specific objectives NOT supervisor-specific

If an item is very clear and meets the RUMBA test, then there is no need to write another objective

29

Objectives: An Example From FWPE for the OT Student

16. Establishes accurate and appropriate plan School—Provides behavioral-based,

measurable OT goals during IEP process

Acute Care—Overall intervention plan is achievable within client’s length of stay

30

Summary Evaluations designed to measure

entry-level competence, NOT level of performance above competency

OT practice examined as a generalist

Evaluations reflect the occupational therapy process

Performance develops over time