The Value of Data The Vital Importance of Accountability American Institutes for Research February 2005

The Value of DataThe Value of Data

The Value of DataThe Vital Importance of Accountability

American Institutes for ResearchFebruary 2005

The Value of DataThe Value of Data2

Federal Uses of NRS DataFederal Uses of NRS Data Develop report to Congress Determine national progress for Program

Assessment Rating Tool (PART) and Government Performance Results Act (GPRA) performance

Assess national and state trends Monitor program outcomes and data quality

Negotiate performance targets with states Determine whether states met prior performance

targets


State Uses of NRS DataState Uses of NRS Data Evaluate local program performance Promote and evaluate local program

improvement efforts Report to legislatures Negotiate state performance targets

with feds


Importance of DataImportance of Data Critical to Federal accountability Supports funding Maintains unique program identity Performance standards: GPRA, Office

of Management and Budget (OMB)


Government Performance and Government Performance and Results Act (GPRA)Results Act (GPRA) Requires annual performance targets tied to

program goals for all Federal programs Adult education’s targets part of ED’s

Strategic Plan Targets—percentage who:

Acquire basic skills to complete level (ABE, ESL)

Transition to postsecondary education Obtain GED Enter employment


GPRA Performance, 2000–GPRA Performance, 2000–20042004










Program Assessment Rating Program Assessment Rating Tool (PART)Tool (PART) OMB review process to enforce GPRA Every program reviewed & scored

annually PART scores must be submitted to

Congress with budget request Secretaries use PART scores to

increase/decrease program $ request Congress uses PART scores to make

appropriations (e.g., de-fund programs)


PART (FY05) Findings for Adult PART (FY05) Findings for Adult Ed. Ed. Adult education’s scores (out of

100%): Program purpose & design (100%) Strategic planning (29%) Program management (67%) Program results (0%)

Summary: “Results Not Demonstrated”


PART “Results” Findings:PART “Results” Findings:

Progress toward long-term goals Meet annual performance goals Demonstrate efficiency and cost

effectiveness Compare favorably to other programs Independent evaluation of program

effectiveness

No No No No No


PARTPART


PARTPART Example from appropriations

committee report: “The Committee recommends no funding

for [this program]. [ED] has not developed performance indicators consistent with the requirements of GPRA…. the Committee has chosen to focus its resources on higher priority programs.” p. 197


From the President’s 2006 From the President’s 2006 Budget:Budget: [Reduced funding for this program] ” … is

consistent with the Administration's goal of decreasing funding for programs with limited impact or for which there is little or no evidence of effectiveness. A PART analysis of the program … produced a Results Not Demonstrated rating. The program was found to have a modest impact on adult literacy, skill attainment and job placement, but data quality problems…. made it difficult to assess the program's effectiveness.”


Independent Program Evaluation Independent Program Evaluation of Effectivenessof Effectiveness One element of PART where we failed,

evaluation study may be coming To show impact we need:

Good assessment data Assessments correctly administered Pre-post scores

Program models—program goals, approach, student participation

Meaningful instructional approach—with standards or a framework


Setting State Performance Setting State Performance StandardsStandards Key to improving national performance Promote continuous improvements Problems resulting from:

High intra-state variance across years Wide variation among states Vastly exceed negotiated level


Intra-state variance across Intra-state variance across yearsyears


Wide variation among statesWide variation among states


Vastly exceed negotiated Vastly exceed negotiated levellevel


Setting State Performance Setting State Performance Standards Standards Compare with past years’ performance Compare to national and median range Equal or exceed actual performance Show continuous improvement State factors

Initiatives, policies, politics Attendance, student variables


Situation similar to 1995 Reauthorization—program will be

evaluated Funding—in a time of huge deficits Renewed interest by the administration in

block grants—workforce focus Need to demonstrate value and identity

of adult education program

Continued Vital Importance of Continued Vital Importance of Accountability DataAccountability Data


DiscussionDiscussion Need for valid and reliable data to

counter PART High quality data

Improve GPRA measures for program Continuous program improvement is

essential Demonstrate program effectiveness

What improvements are needed?

Documents

The Value of Data The Vital Importance of Accountability American Institutes for Research February 2005