Upload
anne-newman
View
217
Download
0
Embed Size (px)
Citation preview
Welcome to Directors’ Program Measure Webinar - Presenter:Jennifer Coffey, Ph.D., OSEP Project Officer, SPDG Program Lead
Click the Person icon to:• Raise Your Hand• Agree/Disagree• Other….
Click the Full Screen: • To maximize
presentation screen
Click This Icon to:- Send private messages- Change text size or chat color
To Post Chat Messages:• Type your message in this
box• Then hit ‘enter’ on your
keyboard to send
Webinar Ground Rules
Mute your phones: - To Mute or Un-Mute Press *6- Please do not put your phones on ‘Hold’
For webinar technical difficulties:- Send email to [email protected]
Q & A Process (audio/chat): - Ask questions in two ways:
1. Audio/Voice2. Type your question in the Chat Pod
Archive Recording, PPT, & Materials- To be posted to; http://signetwork.org/events/108
SAVE THE DATES!
SPDG National Meeting
March 6 & 7, 2012,Washington, DC
OSEP Project Directors’ Conference
July 23-25, 2012, Washington, DC
SPDG National MeetingMarch 5 – MEET UP Night
Upcoming Calls
7
December 1, 2011Time: 3:00-4:30pm ET
Program Measure Directors' Webinar Series #3: Ongoing Technical Assistance and Teacher RetentionJennifer Coffey, PhD, OSEP
Date: January 11, 2012Time: 3:00-4:30pm ET
Adult Learning Principles Carol Trivette, Ph.D.Orelena Hawks Puckett Institute
8
Building a Leadership Support Network from Identification of Need to the Launch--the Oklahoma StoryNovember 7, 2011 12pm EST
The Personnel Improvement Center @ NASDSE and Oklahoma State Department of Education will share how they are building a support network for new special education directors, which includes leadership institutes, webinars with follow-up blog discussions, and mentoring by veteran directors. Network goals include the development and retention of special education directors to ensure that they are able to attract, support and retain highly-qualified, highly-effective special education teachers.
http://click.icptrack.com/icp/relay.php?r=17436393&msgid=408221&act=W7IT&c=572590&destination=https%3A%2F%2Ftadnet.ilinc.com%2Fregister%2Ffmcbsby
10
Professional Learning CommunitiesUpdates
• Disband Secondary Education & Transition PLC (includes Adolescent Literacy)
• Potential Merge – NCRTI – RTI CoP Calls & SPDG RtI/Multi-Tiered Models of Intervention PLC
11
State Grantees Profile – Desktop Share
The Revised SPDG Program Measures: An Overview
Jennifer Coffey, Ph.D. SPDG Program LeadAugust 30, 2011
12
Capturing Performance
Performance Measurement 1: Projects use evidence-based professional development practices to support the attainment of identified competencies.
Performance Measurement 2: Participants in SPDG professional development demonstrate improvement in implementation of SPDG-supported practices over time.
13
Performance Measurement 3: Projects use SPDG professional development funds to provide follow-up activities designed to sustain the use of SPDG-supported practices. (Efficiency Measure)
Performance Measurement 4: Highly qualified special education teachers that have participated in SPDG supported special education teacher retention activities remain as special education teachers two years after their initial participation in these activities.
14
Continuation Reporting
2007 grantees will not be using the new program measures:
Everyone else will have 1 year for practice› Grantees will use the revised measures
this year for their APR› This continuation report will be a pilot
OSEP will learn from this round of reports and make changes as appropriate
15
16
Performance Measurement 2:
Participants in SPDG professional development demonstrate improvement in implementation of SPDG-supported practices over time.
Implementation fidelity Fidelity of implementation is traditionally
defined as “the extent to which the user’s current practice matches the ideal (Loucks, 1983).
17
Citations
Dusenbury, Brannigan, Falco, & Hansen (2003)
Dane & Schneider (1998) O’Donnell (2005) Blase “Innovation Fluency” presentation:
http://signetwork.org/content_pages/154 Mowbray, Holter, Teague & Bybee (2003)
18
From O’Donnell (2005) “All five studies consistently showed
statistically significantly higher outcomes when the program was implemented with greater fidelity.
The studies reviewed here suggest that fidelity of implementation is more probable when an intervention manual is in place that clearly defines the critical components of the intervention and articulates a theory.
Distinctions should be made between measuring fidelity to the structural components of a curriculum intervention and fidelity to the processes that guide its design.”
19
20
EVALUATION DRIVES ERIA’S EVIDENCE-BASED PRACTICES
The Program Guide, a 16-page booklet, explicitly addresses both implementation and intervention practices to guide the design of a site-based program.
The Implementation Rubric is a 10-item instrument which provides a framework for trainers, coaches, site team members, and teachers to evaluate and discuss implementation, fidelity, and next steps.
Some additional tools include: end-of-event training surveys and three-month follow-ups feedback and support from cohort coaches and site team fidelity observations student data
21
ERIA’S EVIDENCE-BASED PRACTICES
The Program Guide articulates a comprehensive set of practices for all stakeholders.
Implementation Practices Intervention Practices
Initial Training
Team-based Site-level Practice and Implementation
Implementation Rubric facilitates self-eval
Ongoing Coaching
Booster Trainings
Implementation Rubric reflection on next steps
The 5 Steps of ERIA
Data-informedDecision-making Screening and
Assessment Progress Monitoring
Tiered Interventions and Learning Supports
Enhanced Literacy Instruction
Measure 2 Methodology
The projects will report on those initiatives that they are reporting on for Program Measure 1
Each initiative should have a fidelity measure that notes the presence or absence of the core features of the innovation/program/system that the initiative is focused on
22
When possible…
Use implementation measures that have already been created› For example – new RTI implementation
measure presented to the RTI PLC› Literacy implementation – Planning and
Evaluation Tool – Revised (PET-R)› Schoolwide Evaluation Tool (SET)› Others?
23
Developing a Fidelity Measure (O’Donnell, 2005)
To develop fidelity criteria, researchers often reported starting with a curriculum profile or analysis that outlined the critical components of the intervention along with an indication of the range of variations for acceptable use. The researcher or developer then outlined acceptable ranges of variation (Songer & Gotwals, 2005).
A component checklist was then developed to record fidelity to these components (Hall & Loucks, 1977).
24
What is “it”? Operationalize
Part of Speech: verb Definition: to define a concept or variable so that it can be measured or expressed quantitatively
Webster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7)Copyright © 2003-2008 Lexico Publishing Group, LLC
The “it” must be operationalized whether it is:
» An Evidence-Based Practice or Program» A Best Practice Initiative or New Framework » A Systems Change Initiative
Practice Profiles » Help Operationalize Practice, Program, and Systems
Features
25
Searching for “It”
Research findings, materials, manuals, and journal articles do not necessarily provide clarity around core intervention elements
Current and new evidence-based practices, frameworks, programs will have a range of operational specificity
Developing clarity around the “it” is critical
26
Practice Profile Defining “it” Through the Development and
Use of Practice Profiles
Guiding Principles identified
Critical Components articulated
Hall and Hord, 2010 Implementing Change: Patterns, Principles, and Potholes (3rd Edition) and Adapted from work of the Iowa Area Education Agency
27
Practice Profile Defining “it” Through the Development and
Use of Practice Profiles
Guiding Principles identified
Critical Components articulated
For each critical component:
Identified gold standard
Identified acceptable variations in practice
Identified ineffective practices and undesirable practices
Hall and Hord, 2010 Implementing Change: Patterns, Principles, and Potholes (3rd Edition) and Adapted from work of the Iowa Area Education Agency
28
????
Have you ever developed or helped to develop a Practice Profile or Innovation Configuration?
Vote Now:» Yes
» No
29
Practice Profiles: Pay Now or Pay Later
Identifies Critical Components Guiding Principles Critical Components Match the Guiding Principles Core Activities to Achieve the Critical Components
For each Critical Component: Identified “gold standard” activities Identified acceptable variations in practice Identified ineffective practices and undesirable practices
Your Implementation Support» Identify and Support Implementation Team» Provide Conceptual Overview and Rationales» Provide Resources, Worksheets, Templates» Facilitate Consensus Building C
apac
ityB
uild
ing
30
Resources for Building Practice Profiles
31
• National Centers• Experts in Your State• National Purveyors• Manuals and Materials• Implementing Districts and
Schools• Other States• Consensus Building in Your State
Example
Problem-Solving Practice Profiles in an RtI Framework
32
RESOURCE - Professional Practices in Problem Solving: Benchmarks and Innovation Configurations
~ Iowa Area Education Agency Directors of Special Education, 1994
Practice Profile Defining “it” Through the Development and
Use of Practice Profiles
Guiding Principles identified
Critical Components articulated
For each critical component:
Identified gold standard
Identified acceptable variations in practice
Identified ineffective practices and undesirable practices
Hall and Hord, 2010 Implementing Change: Patterns, Principles, and Potholes (3rd Edition) and Adapted from work of the Iowa Area Education Agency
33
Practice Profiles
Each Critical Component is a heading
Each level of implementation specifies the activities necessary to operationalize that Critical Component
Critical Component Ideal Implementation
Acceptable Variation
Unacceptable Variation
Unacceptable Variation
Critical Component 1: Description
Description of implementer
behavior
Drastic Mutation
Hall and Hord, 2010, Implementing Change: Patterns, Principles, and Potholes (3rd Edition) and Adapted from work of the Iowa Area Education Agency
34
Professional Problem Solving 9 Critical Components
Parent Involvement
Problem Statement
Systematic Data Collection
Problem Analysis
Goal Development
Intervention Plan Development
Intervention Plan Implementation
Progress Monitoring
Decision Making
Critical Component Ideal Implementation
Acceptable Variation
Unacceptable Variation
Unacceptable Variation
Critical Component 1: Description
Description of implementer
behavior
Professional Practices in Problem Solving: Benchmarks and Innovation ConfigurationsIowa Area Education Agency Directors of Special Education, 1994
35
Professional Problem Solving Parent Involvement as a Critical
ComponentCritical Component Ideal
Implementation
Acceptable Variation
Unacceptable Variation
Unacceptable Variation
Critical Component 1: Description of ParentInvolvement and Rationales for its importance
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” to participate.
Professional Practices in Problem Solving: Benchmarks and Innovation ConfigurationsIowa Area Education Agency Directors of Special Education, 1994
36
Professional Problem Solving Parent Involvement as a Critical
ComponentCritical Component Ideal
Implementation
Acceptable Variation
Unacceptable Variation
Unacceptable Variation
Critical Component 1: Description of ParentInvolvement and Rationales for its importance
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” to participate.
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” not to participate.
Professional Practices in Problem Solving: Benchmarks and Innovation ConfigurationsIowa Area Education Agency Directors of Special Education, 1994
37
Professional Problem Solving Parent Involvement as a Critical
ComponentCritical Component Ideal
Implementation
Acceptable Variation
Unacceptable Variation
Unacceptable Variation
Critical Component 1: Description of ParentInvolvement and Rationales for its importance
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” to participate.
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” not to participate.
Parents are informed of decisions at all decision –making points. But parents are not invited to participate.
Parents are not informed or invited to participate at decision-making points.
Professional Practices in Problem Solving: Benchmarks and Innovation ConfigurationsIowa Area Education Agency Directors of Special Education, 1994
38
Professional Problem Solving Parent Involvement – Critical
Components Critical Component Ideal and/or
AcceptableKnowledge, Skills, and Abilities
DriverAnalysis
Areas ofImpact
(outcomes)
Critical Component: Parent Involvement
Parents are informed at all decision –making points and invited to participate by phone, letter, or email. Parents “choose” to participate.
We know what is necessary to put “it” in place.
This is how we ensure that “it” is in place!
We can prove that we’ve “got it”!
39
Things to Think About Think about your SPDG effort and your
involvement and guidance at the State, District, and School levels.
Currently, our SPDG work is well operationalized ? ….At the Classroom level
» _Strongly Agree _Agree __Disagree __Strongly Disagree
…At the district level
…At the regional level
…At the State level
40
Michigan’s Practice Profile: Building Leadership Team Example
41
42
CALIFORNIA’S EVALUATION TOOL:IMPLEMENTATION RUBRIC
The 10 items are intervention practices-focused mostly, with site team and fidelity items
The overall tool and process of how the rubric isused drives the implementation practices Self-evaluate and reflect on learning and
implementation. Shared with coaches and trainers to guide activities Evaluates the fidelity of implementation of both the PD
model and the interventions
Former 26-item, 3-point ERIA Checklist lacked the specificity to be meaningful and useful.
43
IMPLEMENTATION RUBRIC, ADAPTED FROM “GOAL ATTAINMENT SCALES” Amy Gaumer Erickson and Monica Ballay presented
“goal attainment scales” on a June 17 SIG Network webinar: http://www.signetwork.org/content_pages/78
Rubric explicitly describes 5 implementation levels for each of 10 items: Levels 1, 2, and 3 reflect the “Not started,” “In
progress,” and “Achieved” implementation levels of former checklist.
Levels 4 and 5 detail concrete steps towards optimal implementation, beyond the basics.
Each implementation level for each item is explicitly described, building more meaning into the tool than our previous checklist format allowed.
44
IMPLEMENTATION RUBRIC EXCEL FILE:MULTI-YEAR TRACKING AND AUTOMATED REPORTS
The same file is used in all three years of ERIA,reporting both the trend and most-recent entries.
45
ERIA on the Web:http://calstat.org/effectivereading.html
Li Walter: [email protected]
Alan Wood: [email protected] (707) 287-0054
Other Research on Implementation Fidelity
Observations may be crucial because teachers are known to be biased in their reports (Hansen and McNeal, 1999).
Given the frequency with which adaptations are observed in research and practice, program developers need to anticipate how and when teachers will modify programs and develop guidelines and recommendations to ensure program goals are met (Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005)
46
• The project will set its own benchmarks for professional development participants 1 year into training/assistance, 2 yrs in, 3 yrs in, 4 yrs in • For example: 1 yr benchmark = 40% of core
features in place, 4 yr benchmark = 80% of features in place
The project will then determine what percentage of participants they expect to reach this benchmark (e.g., 80% of participants)
a.Participants could be individual teachers (if working with just a few teachers or other type of professional per school or district) or could be a school (if working on a school-wide basis, such as RTI or PBIS)
47
Self-assessment is acceptable, but projects will need to sample from the group to validate the self-assessment
a.For example, if 15 schools were being measured someone from the project would observe at least 3 (1/5th) of the schools and compare their assessment with the self-assessment
A baseline wouldn’t be necessary
48
Questions?
What kind of guidance would be helpful to you?
What challenges do you foresee in capturing these data?
49