Assessment Systems as Learning Communities AACTE/NCATE Institutional Orientation April 3, 2008 Hilda Rosselli, Dean Western Oregon College of Education

Assessment Systems as Learning Communities

AACTE/NCATE Institutional Orientation

April 3, 2008

Hilda Rosselli, DeanWestern Oregon College of

[email protected]

AACTE/NCATE Institutional Orientation

April 3, 2008

Hilda Rosselli, DeanWestern Oregon College of

[email protected]

April 3-6, 2008 H.Rosselli (AACTE/NCATE) 2

A Recursive Process for Candidates

TeacherWork

Sample

Candidates AssessPK-12 Student Learning

Candidates ReviewResults of TWS

Candidates Reflect and Plan Next Steps

Assessment Systems as Learning Communities

Candidates Teachtheir Students


A Recursive Process for Faculty

Faculty AssessCandidate Learning

Faculty ReviewResults of Key Assessments

DARE

Faculty TeachCandidates

Faculty ReviewAnd Refine


• Small public liberal arts college (5,000 students)

• Historically known for its teacher preparation programs (1856)

• License around 300 teachers per year

30+ faculty in teacher preparation Undergrad, MAT (two formats), SPED,

and MSinEd NCATE accredited for over 50 years

Context


Connecting Teaching & Learning• Home of Teacher

Work Sample Methodology

• Del Schalock, TRI mentor to the College

• Leadership role in Oregon teacher preparation



Foci for Faculty

Refinement of prompts and rubrics Inter-rater reliability and validity for

scorers of TWSM Systematic linkage of TWSM elements to

state and national standards Formalized rules for passage through

gates based on candidate performance; Easy access to aggregated data for use in

reporting, research and program improvement


Our collective tasks… Maximize existing data already being

collected; Select the most compelling types of data for

aggregation; Document the rubrics and decision making

rules for each assessment; Revisit alignment with CF, INTASC, TSPC,

and WOU proficiencies; Create a predictable cycle of compilation,

analysis, and review; Develop relational databases to support

deeper levels of analysis;


Our collective tasks… Capitalize on existing and emerging

technologies to support the system; Engage staff, faculty, administrators, and

advisory boards throughout the process; Synchronize efforts with any institutional

assessment efforts when possible; Support faculty engagement in scholarship

related to assessment; Institutionalize a culture in which data are

regularly reviewed and used for program verification and improvement


Process• Work Groups

Dispositions, Diversity, TWSM, Portfolios, Policies, Data Management System, Follow up Studies, Field Experiences

• Leadership Team• College Days• Communication• Partnership w/Computing Services and

Registrar


BASIC PARAMETERS OF A DATA MANAGEMENT SYSTEM FOR THE

INITIAL LICENSING OF TEACHERS AT WOU

PERFORMANCE/ MEASUREMENT FOCUS

GATE I Program

Admission

GATE II

Short-Term Supported Teaching

Experiences

GATE III

Admission to Long-Term Student

Teaching

GATE IV Recommendation for INITIAL Licensure

Knowledge/understanding (CBEST, ORELA, PRAXIS, GPA, coursework)

X

X

X

Skills/performance in school settings (Formative)

X

X

Dispositions/interview data

X X X X

Impact on student learning (TWSM)

X X

ELECTRONIC INFORMATION MANAGEMENT SYSTEM

Candidate ID/demographics

X

Placement contextual ID/demographics

X X X

Exit surveys

X

Follow up (Alumni & Employer surveys)

2 years after


Flowchart

NCATE Domain

Program Proficiency

Element Assessment

Data RuleYes

No Flag

Leads to

Tests

Allows us to answer

Raises questions

about

leads to

estaablishes

raises the need for

raises questions

forces

invites

RESULTS

IN

COE Data

Management

System

Quality

Assurance

Program

Accountability

Conceptual

Framework

Current

Practices

Current

Policies

ASSUMPTION

Knowledge Skills

Dispositions

informs and aligns

with a College's

Past

"Tweaking"

forces

review of

Revisions to

Conceptual

Framework

New

Practices

New

Policies

Specific

Assignments

Ajustments

within

programs

Adjustments

Across

Programs

Increased

focus

New clerical

responsibilities

New Issues

Confidentiality

Flags

Role of

Portfolios

Finetuning

TWSM

rubrics

Reliability &

Validity of

Assessment

Protocals

What we

can say

about our

graduates

What

changes

need to

occur in

programs

Data

Reporting

Research

Program

Improvement

Diversity Dispositions

Impact on

student

learning

Observation

forms

Proficiencies

Deaf Ed

Quality

Assurance and

Program

Accountability

DISPOSITIONS

PROFESSIONAL

KNOWLEDGE

PEDAGOGICAL

KNOWLEDGE

CONTENT

KNOWLEDGE

8.Philosophy,

Best Practice,

& Reflection

9.Professional

Commitment &

Dispositions

10. Collaboration,

Partnerships &

Leadership

2. Human

Development

& Learning

3. Diversity &

Exceptionality

4. Assessment

& Instruction

5. Classroom

Climate

7.Technology

6. Literacy Dev.

&

Communication

1.Content

Knowledge

& Pedagogy

NCATE

DOMAINS

WOU

PROFICIENCIES

Depth & Breadth*

Integration 1b

Alignment 4b

Curriculum Modific.-3f, 3a

ASSESSMENT ELEMENTS

Reflects strong link with impact on

student learning as reflected in TWSM.

Constructive Communication 2.1

Multi-Disciplinary Teaming 4.3

Assistance & Support

Learning Community

Active Participation 4.3

Professional Behavior 5a

Support Learning

Community 5b

Professional Communication 2g

Self-direction ?2.4?, 4a

Reflective Practitioner* 3.2

Philosophy of Education

Educational Research* 3d

Responsibility* 3.2

Inclusion*-1f,2c, 4a

Awareness

Advocacy

Learning Theories*

Human Development Models 4a

Applications 3a, 3d

Benchmarks & Assessments

Literature

Language Development

2nd Language* -2b, 2e

Cultural Inflluence

Facets of Literacy

Literacy Programs

Assessment Tools 4a,b

Goals & Standards

Planning--1a-g, 2j, 3c,e,

5a, c, d

Implementation*

Time Management 1g,2h,i

Classroom Assistants*2k

Human Dignity*-2a,e

Physical Environment

Student Behavior*-2b,d,e

Classroom Climate 5c

Social & Ethical Issues*

Operations & Concepts 1e

Designing Learning Environments

Evaluating/Selecting Software*

Assessment Evaluation

Undergraduate Program Gate 4

Recommendation for Initial Licensure

Professional Practice

GATE 4

Recommendation for Inital Licensure

2 Work

samples

Completion

of Student

Teaching

Y

N

Summative

Evaluationapproval

by both

Tests

Required

Coursework

Y

N

RegistrarApproval /

maintain

GPA

Y

N

Field Book

EvaluationsScore of

proficient

Y

N

Praxis Passing

Disabilities* - 2c, 2e, 2g

Human Dignity*-2a,e

Classroom Assistants*2k

Integration*-1f,2c, 4a

Graduation and

Recommendation

for Initial

Licensure


TWS as a Basis for Reflective Learning Community Dialogue

Candidate with Students Candidate with Candidates Candidate with Mentor Teacher Candidates with Faculty Faculty with Mentor Teachers Faculty to Faculty


A Continuous Teaching CycleA Continuous Teaching Cycle

PreAssess

Post Assess

Analyze and reflect

Plan

Instruct


D.A.R.E.

Data Analysis Review Event-held annually

Review what we are learning about our students, our programs, ourselves…

Use what we are learning to make informed decisions about our programs…

Celebrate our efforts and our community.


Why look at data at all?

Focus on what we know about our students

Focus on what is happening to our students while they are with us

Focus on our students after they leave us Plan based on what we learn Share information within and across areas Ask questions of ourselves Fine-tune our assessments and processes


Format for presenters

What question/s was investigated and who were the participants?

What process was used to investigate the question/s?

What were the findings? What implications do you see

resulting from the findings? Next steps (if applicable)


Typical D.A.R.E. agenda

TWSM Protocol Changes-Adele Schepige COE Faculty Survey on Inclusivity-Chloe Myers MS in Ed Alumni Surveys-Linda Stonecipher Candidate Exit Surveys-Connie Wildfang CF Walk the Talk-Mark Girod Clinical Formative Assessments-Sue Thompson Employer Satisfaction and Alumni-Hilda

Rosselli


Beyond Teacher Prep Table Talks

WOU Admission Surveys-David McDonald WOU Faculty Climate Surveys-Hilda Rosselli WOU Freshmen Retention Rates-IR COE Enrollment Management Results-Tiffany

Smith HPE Student Survey II-Jessica Henderson National College Health Assessment-Peggy

Pedersen WOU Advising Survey-Karen Sullivan-Vance RC/RCD Follow Up Data-Hank Bersani


Entering Candidate Data

C B S T M T H

7 0 . 06 5 . 06 0 . 05 5 . 05 0 . 04 5 . 04 0 . 03 5 . 0

F a l l 2 0 0 5 U n d e r g r a d u a t e C o h o r t

C B E S T M a t h S c o r e s

n = 4 6

1 0

8

6

4

2

0

S t d . D e v = 9 . 2 0

M e a n = 5 2 . 8

N = 4 6 . 0 0

C B S T W R T G

8 0 . 07 0 . 06 0 . 05 0 . 04 0 . 0

F a l l 2 0 0 5 U n d e r g r a d u a t e C o h o r t

C B E S T W r i t i n g S c o r e s

n = 4 6

4 0

3 0

2 0

1 0

0

S t d . D e v = 8 . 7 5

M e a n = 4 4 . 3

N = 4 6 . 0 0


Table 7.1Formative Evaluation Component: Selected Items

Teacher Candidate Performance over TimeUndergraduate Program: Fall 2005 Cohort

Mean Scores & Standard Deviation (sd)

2.983.14 3.12

2.903.14 3.103.12 3.16 3.16

1

1.5

2

2.5

3

3.5

4

UG Fall 2005Gate: Term II

UG Fall 2005Gate: Term III

UG Fall 2005Gate: Term IV

Planning for InstructionItem A

Planning For InstructionItem D

Engaging StudentsItem C

(.44) (.55)(.53) (.45) (.45) (.51) (.39) (.46) (.37)

N = 50 Teacher Candidates from the Fall 2005 Undergraduate Cohort

Score Rubric1 = Needs Improvement2 = Beginning3 = At Required Level4 = Exceeds Required Level

Clinical Ratings by Item Across Gates


Longitudinal Data TrendsUse Standards/Benchmarks for Long-term Curriculum

Planning

76%

78%

94%

80%

93%

64%

98%

100%

100%

24%

23%

6%

20%

8%

36%

3%

0%

0%

0% 50% 100% 150%

SP03 (N=97)

F03 (N=69)

W04 (N=50)

SP04 (N=67)

F04 (N=51)

W05 (N=42)

SP05 (N=38)

F05 (N=47)

W06 (N=40)

Poor-Fair

Good-Excellent


MSEd Alumni Perception of Program Impact on Content Pedagogy

4

8

13

11

1

1 = not at all 2 = slightly 3 =moderately

4 =significantly

No Response

Rating

n = 37


Role of External Consortium Previews prototypes Reviews results presented at DARE Recommended that faculty address

all items with a combined score of less than 80% for Good to Excellent.

Reviews faculty reflections and recommendations

Includes LAS representation


2007 Regional Accreditation

“The leaders in the College of Education seek feedback on their performance from a wide variety of sources--and the annual Data Analysis Review Event provides an opportunity for all faulty to become engaged in the process of analyzing and interpreting the results. Significant policy and procedural changes have resulted from the Events held during the past three years.”


2007 NCATE comment

“The WOU Teacher Work Sample methodology has long provided an effective way to monitor candidates’ ability to analyze the teaching/learning context, set goals, plan and assess instruction, adapt instruction to meet individual needs of students, and reflect on their own work. The process has remained basically the same, while methods of assessment have changed, each based on perceived need, and with careful thought toward excellence.”


2007 NCATE comment

“The COE actively involves tenure-track, term, and adjunct faculty in improving programs for the purpose of ensuring that candidates are prepared to help students learn.”


Closing Thought

“Among the factors that appear to be critical to a profession-wide change process, several seem particularly important:

• coherence in standards, • clarity about curriculum, and • commitment to act based on knowledge

about what appears to be effective.” p. 479

Preparing Teachers for a Changing World

Documents

Assessment Systems as Learning Communities AACTE/NCATE Institutional Orientation April 3, 2008 Hilda Rosselli, Dean Western Oregon College of Education