View
218
Download
1
Category
Tags:
Preview:
Citation preview
Assessment Systems as Learning Communities
AACTE/NCATE Institutional Orientation
April 3, 2008
Hilda Rosselli, DeanWestern Oregon College of
Educationrossellh@wou.edu
AACTE/NCATE Institutional Orientation
April 3, 2008
Hilda Rosselli, DeanWestern Oregon College of
Educationrossellh@wou.edu
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 2
A Recursive Process for Candidates
TeacherWork
Sample
Candidates AssessPK-12 Student Learning
Candidates ReviewResults of TWS
Candidates Reflect and Plan Next Steps
Assessment Systems as Learning Communities
Candidates Teachtheir Students
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 3
A Recursive Process for Faculty
Faculty AssessCandidate Learning
Faculty ReviewResults of Key Assessments
DARE
Faculty TeachCandidates
Faculty ReviewAnd Refine
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 4
• Small public liberal arts college (5,000 students)
• Historically known for its teacher preparation programs (1856)
• License around 300 teachers per year
30+ faculty in teacher preparation Undergrad, MAT (two formats), SPED,
and MSinEd NCATE accredited for over 50 years
Context
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 5
Connecting Teaching & Learning• Home of Teacher
Work Sample Methodology
• Del Schalock, TRI mentor to the College
• Leadership role in Oregon teacher preparation
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 6
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 7
Foci for Faculty
Refinement of prompts and rubrics Inter-rater reliability and validity for
scorers of TWSM Systematic linkage of TWSM elements to
state and national standards Formalized rules for passage through
gates based on candidate performance; Easy access to aggregated data for use in
reporting, research and program improvement
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 8
Our collective tasks… Maximize existing data already being
collected; Select the most compelling types of data for
aggregation; Document the rubrics and decision making
rules for each assessment; Revisit alignment with CF, INTASC, TSPC,
and WOU proficiencies; Create a predictable cycle of compilation,
analysis, and review; Develop relational databases to support
deeper levels of analysis;
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 9
Our collective tasks… Capitalize on existing and emerging
technologies to support the system; Engage staff, faculty, administrators, and
advisory boards throughout the process; Synchronize efforts with any institutional
assessment efforts when possible; Support faculty engagement in scholarship
related to assessment; Institutionalize a culture in which data are
regularly reviewed and used for program verification and improvement
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 10
Process• Work Groups
Dispositions, Diversity, TWSM, Portfolios, Policies, Data Management System, Follow up Studies, Field Experiences
• Leadership Team• College Days• Communication• Partnership w/Computing Services and
Registrar
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 11
BASIC PARAMETERS OF A DATA MANAGEMENT SYSTEM FOR THE
INITIAL LICENSING OF TEACHERS AT WOU
PERFORMANCE/ MEASUREMENT FOCUS
GATE I Program
Admission
GATE II
Short-Term Supported Teaching
Experiences
GATE III
Admission to Long-Term Student
Teaching
GATE IV Recommendation for INITIAL Licensure
Knowledge/understanding (CBEST, ORELA, PRAXIS, GPA, coursework)
X
X
X
Skills/performance in school settings (Formative)
X
X
Dispositions/interview data
X X X X
Impact on student learning (TWSM)
X X
ELECTRONIC INFORMATION MANAGEMENT SYSTEM
Candidate ID/demographics
X
Placement contextual ID/demographics
X X X
Exit surveys
X
Follow up (Alumni & Employer surveys)
2 years after
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 12
Flowchart
NCATE Domain
Program Proficiency
Element Assessment
Data RuleYes
No Flag
Leads to
Tests
Allows us to answer
Raises questions
about
leads to
estaablishes
raises the need for
raises questions
forces
invites
RESULTS
IN
COE Data
Management
System
Quality
Assurance
Program
Accountability
Conceptual
Framework
Current
Practices
Current
Policies
ASSUMPTION
Knowledge Skills
Dispositions
informs and aligns
with a College's
Past
"Tweaking"
forces
review of
Revisions to
Conceptual
Framework
New
Practices
New
Policies
Specific
Assignments
Ajustments
within
programs
Adjustments
Across
Programs
Increased
focus
New clerical
responsibilities
New Issues
Confidentiality
Flags
Role of
Portfolios
Finetuning
TWSM
rubrics
Reliability &
Validity of
Assessment
Protocals
What we
can say
about our
graduates
What
changes
need to
occur in
programs
Data
Reporting
Research
Program
Improvement
Diversity Dispositions
Impact on
student
learning
Observation
forms
Proficiencies
Deaf Ed
Quality
Assurance and
Program
Accountability
DISPOSITIONS
PROFESSIONAL
KNOWLEDGE
PEDAGOGICAL
KNOWLEDGE
CONTENT
KNOWLEDGE
8.Philosophy,
Best Practice,
& Reflection
9.Professional
Commitment &
Dispositions
10. Collaboration,
Partnerships &
Leadership
2. Human
Development
& Learning
3. Diversity &
Exceptionality
4. Assessment
& Instruction
5. Classroom
Climate
7.Technology
6. Literacy Dev.
&
Communication
1.Content
Knowledge
& Pedagogy
NCATE
DOMAINS
WOU
PROFICIENCIES
Depth & Breadth*
Integration 1b
Alignment 4b
Curriculum Modific.-3f, 3a
ASSESSMENT ELEMENTS
Reflects strong link with impact on
student learning as reflected in TWSM.
Constructive Communication 2.1
Multi-Disciplinary Teaming 4.3
Assistance & Support
Learning Community
Active Participation 4.3
Professional Behavior 5a
Support Learning
Community 5b
Professional Communication 2g
Self-direction ?2.4?, 4a
Reflective Practitioner* 3.2
Philosophy of Education
Educational Research* 3d
Responsibility* 3.2
Inclusion*-1f,2c, 4a
Awareness
Advocacy
Learning Theories*
Human Development Models 4a
Applications 3a, 3d
Benchmarks & Assessments
Literature
Language Development
2nd Language* -2b, 2e
Cultural Inflluence
Facets of Literacy
Literacy Programs
Assessment Tools 4a,b
Goals & Standards
Planning--1a-g, 2j, 3c,e,
5a, c, d
Implementation*
Time Management 1g,2h,i
Classroom Assistants*2k
Human Dignity*-2a,e
Physical Environment
Student Behavior*-2b,d,e
Classroom Climate 5c
Social & Ethical Issues*
Operations & Concepts 1e
Designing Learning Environments
Evaluating/Selecting Software*
Assessment Evaluation
Undergraduate Program Gate 4
Recommendation for Initial Licensure
Professional Practice
GATE 4
Recommendation for Inital Licensure
2 Work
samples
Completion
of Student
Teaching
Y
N
Summative
Evaluationapproval
by both
Tests
Required
Coursework
Y
N
RegistrarApproval /
maintain
GPA
Y
N
Field Book
EvaluationsScore of
proficient
Y
N
Praxis Passing
Disabilities* - 2c, 2e, 2g
Human Dignity*-2a,e
Classroom Assistants*2k
Integration*-1f,2c, 4a
Graduation and
Recommendation
for Initial
Licensure
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 15
TWS as a Basis for Reflective Learning Community Dialogue
Candidate with Students Candidate with Candidates Candidate with Mentor Teacher Candidates with Faculty Faculty with Mentor Teachers Faculty to Faculty
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 16
A Continuous Teaching CycleA Continuous Teaching Cycle
PreAssess
Post Assess
Analyze and reflect
Plan
Instruct
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 17
D.A.R.E.
Data Analysis Review Event-held annually
Review what we are learning about our students, our programs, ourselves…
Use what we are learning to make informed decisions about our programs…
Celebrate our efforts and our community.
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 18
Why look at data at all?
Focus on what we know about our students
Focus on what is happening to our students while they are with us
Focus on our students after they leave us Plan based on what we learn Share information within and across areas Ask questions of ourselves Fine-tune our assessments and processes
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 19
Format for presenters
What question/s was investigated and who were the participants?
What process was used to investigate the question/s?
What were the findings? What implications do you see
resulting from the findings? Next steps (if applicable)
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 20
Typical D.A.R.E. agenda
TWSM Protocol Changes-Adele Schepige COE Faculty Survey on Inclusivity-Chloe Myers MS in Ed Alumni Surveys-Linda Stonecipher Candidate Exit Surveys-Connie Wildfang CF Walk the Talk-Mark Girod Clinical Formative Assessments-Sue Thompson Employer Satisfaction and Alumni-Hilda
Rosselli
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 21
Beyond Teacher Prep Table Talks
WOU Admission Surveys-David McDonald WOU Faculty Climate Surveys-Hilda Rosselli WOU Freshmen Retention Rates-IR COE Enrollment Management Results-Tiffany
Smith HPE Student Survey II-Jessica Henderson National College Health Assessment-Peggy
Pedersen WOU Advising Survey-Karen Sullivan-Vance RC/RCD Follow Up Data-Hank Bersani
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 22
Entering Candidate Data
C B S T M T H
7 0 . 06 5 . 06 0 . 05 5 . 05 0 . 04 5 . 04 0 . 03 5 . 0
F a l l 2 0 0 5 U n d e r g r a d u a t e C o h o r t
C B E S T M a t h S c o r e s
n = 4 6
1 0
8
6
4
2
0
S t d . D e v = 9 . 2 0
M e a n = 5 2 . 8
N = 4 6 . 0 0
C B S T W R T G
8 0 . 07 0 . 06 0 . 05 0 . 04 0 . 0
F a l l 2 0 0 5 U n d e r g r a d u a t e C o h o r t
C B E S T W r i t i n g S c o r e s
n = 4 6
4 0
3 0
2 0
1 0
0
S t d . D e v = 8 . 7 5
M e a n = 4 4 . 3
N = 4 6 . 0 0
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 23
Table 7.1Formative Evaluation Component: Selected Items
Teacher Candidate Performance over TimeUndergraduate Program: Fall 2005 Cohort
Mean Scores & Standard Deviation (sd)
2.983.14 3.12
2.903.14 3.103.12 3.16 3.16
1
1.5
2
2.5
3
3.5
4
UG Fall 2005Gate: Term II
UG Fall 2005Gate: Term III
UG Fall 2005Gate: Term IV
Planning for InstructionItem A
Planning For InstructionItem D
Engaging StudentsItem C
(.44) (.55)(.53) (.45) (.45) (.51) (.39) (.46) (.37)
N = 50 Teacher Candidates from the Fall 2005 Undergraduate Cohort
Score Rubric1 = Needs Improvement2 = Beginning3 = At Required Level4 = Exceeds Required Level
Clinical Ratings by Item Across Gates
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 24
Longitudinal Data TrendsUse Standards/Benchmarks for Long-term Curriculum
Planning
76%
78%
94%
80%
93%
64%
98%
100%
100%
24%
23%
6%
20%
8%
36%
3%
0%
0%
0% 50% 100% 150%
SP03 (N=97)
F03 (N=69)
W04 (N=50)
SP04 (N=67)
F04 (N=51)
W05 (N=42)
SP05 (N=38)
F05 (N=47)
W06 (N=40)
Poor-Fair
Good-Excellent
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 25
MSEd Alumni Perception of Program Impact on Content Pedagogy
4
8
13
11
1
1 = not at all 2 = slightly 3 =moderately
4 =significantly
No Response
Rating
n = 37
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 26
Role of External Consortium Previews prototypes Reviews results presented at DARE Recommended that faculty address
all items with a combined score of less than 80% for Good to Excellent.
Reviews faculty reflections and recommendations
Includes LAS representation
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 27
2007 Regional Accreditation
“The leaders in the College of Education seek feedback on their performance from a wide variety of sources--and the annual Data Analysis Review Event provides an opportunity for all faulty to become engaged in the process of analyzing and interpreting the results. Significant policy and procedural changes have resulted from the Events held during the past three years.”
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 28
2007 NCATE comment
“The WOU Teacher Work Sample methodology has long provided an effective way to monitor candidates’ ability to analyze the teaching/learning context, set goals, plan and assess instruction, adapt instruction to meet individual needs of students, and reflect on their own work. The process has remained basically the same, while methods of assessment have changed, each based on perceived need, and with careful thought toward excellence.”
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 29
2007 NCATE comment
“The COE actively involves tenure-track, term, and adjunct faculty in improving programs for the purpose of ensuring that candidates are prepared to help students learn.”
April 3-6, 2008 H.Rosselli (AACTE/NCATE) 30
Closing Thought
“Among the factors that appear to be critical to a profession-wide change process, several seem particularly important:
• coherence in standards, • clarity about curriculum, and • commitment to act based on knowledge
about what appears to be effective.” p. 479
Preparing Teachers for a Changing World
Recommended