The Process we Used and What we Learned….

Renee Bradley, Judy Shanley – OSEPHerb Baum, Danielle Schwarzmann – ICF Macro

2009 OSEP Project Director’s MeetingJuly 20th, 2009

05/03/23 1

Thank you to those projects who participated

It’s never too early to think about Performance Measurement… Performance Measures: Under the Government Performance and

Results Act of 1993 (GPRA), the Department has established a set of performance measures, including long-term measures, that are designed to yield information on various aspects of the effectiveness and quality of the Technical Assistance and Dissemination to Improve Services and Results for Children with Disabilities program. These measures focus on the extent to which projects provide high quality products and services, the relevance of project products and services to educational and early intervention policy and practice, and the use of products and services to improve educational and early intervention policy and practice. The grantee will be required to provide information related to these measures. The grantee also will be required to report information on the project’s performance in annual reports to the Department (34 CFR 75.590).

Applications, work scope, reporting requirements

05/03/23 2

Provides aggregate picture of our performance

Affects potentially, future program funding Supports profession of TA&D Enhances legitimacy of the field of TA&D

05/03/23 3

Annual◦ Quality, usefulness, relevance, cost (efficiency)

Long-term◦ Implementation of evidence-based practices◦ Implementation of model demonstrations

05/03/23 4

5

Annual MeasurePercentage of Special Education Technical Assistance

and Dissemination (TA&D) Products and services deemed to be of high quality by an independent (science) review panel (annual).

Substance:◦ Does the product or service reflect an evidence-based

approach or one grounded in current legislation or policy?

Communication:◦ Is the presentation of the product or service

description clear, well-formatted, organized?

05/03/23

6

Annual MeasurePercentage of Special Education Technical Assistance and

Dissemination (TA&D) Products and services deemed to be of useful by an independent (science) review panel (annual).

Ease:◦ Is the product or service description easily understood,

providing guidance and direction? Replicability:

◦ Will the product or service eventually be used by the target group to achieve the benefit intended?

Sustainability:◦ Will the product or service eventually be used in more

than one setting and over time?

05/03/23

7

Annual MeasurePercentage of Special Education Technical Assistance and

Dissemination (TA&D) Products and services deemed to be of relevant by an independent (science) review panel (annual).

Need:◦ Does the product or service solve an important

problem or address a critical issue? Pertinence:

◦ Does the product or service relate directly to the problem or issue facing the targeted group?

Reach:◦ Does the product or service apply to diverse

populations within the target group?

05/03/23

8

OMB requests each Federal program to provide information on cost.◦ Cost is defined as the $ per unit of output.◦ Cost includes labor and other direct costs.◦ Cost includes only those expenses incurred by

Federal project.◦ For TA&D this is the cost of producing the ‘best’

practice/service.

05/03/23

9

Implemented this year for TA&D program for the first time

Twenty-two TA&D projects and 10 State Deaf-blind projects randomly selected

Two Long-term Measures: ◦ Implementation of Evidence-Based Practices◦ Implementation of Model-Demonstration Projects

05/03/23

10

Percentage of school districts and service agencies receiving Special-Education TA&D services regarding scientifically or evidenced-based practices for infants, toddlers, children and youth with disabilities that implement those practices (long-term).

05/03/23

11

The percentage of TA&D projects responsible for developing models that identify, implement, and evaluate effective models (long-term).

05/03/23

05/03/23 12

Dr. Herbert Baum,Danielle Schwarzmann

ICF Macro

Selection◦ Random Selection

Using placemat◦ Additional sites suggested by OSEP to ensure

Evidence—based practice areas were represented Data Collection

◦ Initial e-mail◦ Follow-up/ Reminder e-mails◦ Phone calls when necessary

05/03/23 13

1. E-mail requesting ‘best’ product/service (including costs), and list of services/products

2. Randomly selected a product/service as a ‘typical’ submission

3. Requested the program complete the ‘typical’ description

05/03/23 14

Products/services are evaluated for 3 criteria1. Quality

Substance Communication

2. Relevance Need Pertinence Reach

3. Usefulness Ease Likelihood of Use Replicability

05/03/23 15

OMB requests each federal program to provide information on cost.

Cost is defined as the $ per unit of output. For TA&D this is the cost of producing the

‘best’ practice/service.

05/03/23 16

TA&D centers submitted forms indicating their practices/programs and where they were being implemented

ICF Macro chose a practice, and a place of implementation to complete a form

05/03/23 17

Best Product/Service Description Typical Product/Service Description

Product Service Have not submitted

Response Rate Product Service Have not

submittedResponse

Rate

TA&D 17 5 0 100% 12 4 6 72%

Deaf/Blind 10 0 0 100% 7 2 1 90%

List of Products/Services Best Cost Measure 1.1

Submitted Have not Submitted

Response Rate Submitted Have not

SubmittedResponse

Rate Submitted Have not Submitted

Response Rate

TA&D 20 2 91% 17 5 77% 22 0 100%

Deaf/Blind 9 1 90% 10 0 100%

05/03/23 18

Consider using electronic submissions - may not need to mail hard copies You save money on shipping Decreases response time Trees are saved

Submit accessible products - Diminish use of copywritten products as exemplars

Products – not links to websites must be submitted

Talk with your Project Officer and TA recipients about the Program Evaluation

05/03/23 19

April◦ Final review of methodology by OSEP◦ Preparation of protocol materials

May◦ Select sample of projects◦ Send requests to projects

05/03/23 20

June◦ Obtain description of ‘best” and “typical””

product and service◦ Obtain lists of products and services◦ Obtain products◦ Obtain cost data

July - Review by expert panels August – Generate Measures September – Report finings and

recommendations to OSEP

05/03/23 21

Collect feedback from 2009 samples and use to improve process

Share results Discuss with grantees in kick-off meetings

and during the year Explore ways to integrate content from

annual continuation reports into the annual program evaluation

Learn from other Federal agencies Support and engage OSEP Project Officers

05/03/23 22

Comments, Questions, Concerns?◦ What could we do to improve our process?◦ What could we do to enhance definitions and

instructions?◦ What can we do to make life easier for you?

05/03/23 23