Results-Based Accountability

Using data to change policy and

program practice

Joseph McLaughlinResearch and

Evaluation Director

Lindsey LathropAssistant Director

Success Boston Initiative

A case study on measuring

systemic and organizational

impact

What is Success Boston?

• Success Boston is a cross-sector collaboration between The Boston Foundation, City of Boston, Boston Public Schools (BPS), nonprofit organizations, and higher education institutions (38 schools)

• The initiative’s work is organized along a “Getting Ready, Getting In, and Getting Through” framework

• Common goal to increase the number of BPS graduates who earn 2 or 4-year degrees, particularly among low income students of color

Graduation Rate Outcomes and Goals

• 2008 Study- “Getting to the Finish Line” revealed that only 35% of BPS college enrollees from the Class of 2000 graduated by 2007- seven years after high school.

• Success Boston initiative launched with two specific goals:

– Increase the graduation rate for the Class of 2009 to 52% (150% of Class of 2000)

– Double the graduation rate for the Class of 2011 (200% of Class of 2000)

• Transition coaching program piloted with the Class of 2009

Performance Measures(National Student Clearinghouse ‘NSC’ is primary data source)

• Enrollment rates

• Remedial course-taking rates

• Persistence rates

• Graduation rates

• College coaching impact measures (quasi-experimental studies)

*All measures are disaggregated for demographic and schooling subgroups

Cumulative College Enrollment Rates,

Seven Years after Graduation, NSC Data

67.2

73.8

77.0

60

62

64

66

68

70

72

74

76

78

80

2000 2003 2005

Pe

r C

en

t

Class

Six Year Graduation Rates of College Enrollees Who Enrolled in College in First Year after Graduation

40.6

46.547.4

49.2

30

32

34

36

38

40

42

44

46

48

50

2000 2003 2005 2006

Pe

r C

en

t

Class

One-Year College Persistence Rates,All and by Type of College Attended (in percent)

Graduating Class

(A)

AllColleges

(B)

Two YearCollege

(C)

Four YearCollege

2001 78.1 55.4 86.5

2003 81.6 61.0 87.5

2007 81.3 61.3 87.3

2008 82.5 66.5 87.5

Percentage Point Change +4.4 +11.1 +1.0

One- and Two-Year Persistence Rates of Class of 2009

BPS Graduates at Seven Selected Colleges, by Success

Boston Coaching Participation

Enrolled in Success Boston

Not in SuccessBoston Difference

Group

1-Yr Persistence

Rate

2-Yr Persistence

Rate

1-Yr Persistence

Rate

2-Yr Persistence

Rate

1-Yr Persistence

Rate

2-Yr Persistence

Rate N

All 86.4 73.4 66.0 49.9 +20.4 +23.5 711

Male 83.3 66.7 62.5 46.6 +20.9 +20.1 343

Female 88.1 77.1 70.0 53.6 +18.1 +23.5 368

Black 90.3 77.4 64.8 49.6 +25.6 +27.8 306

Hispanic 82.6 66.3 58.6 42.7 +24.0 +23.6 243

Estimated Percentage Point Impacts of Participation in

Success Boston Programs on the One Year and Two Year

Persistence Rates of BPS Graduates from the Class of 2009,

All and by Gender

Group

(A)

One YearPersistence

Rate

(B)

Sig.Level

(C)

Two YearPersistence

Rate

(D)

Sig.Level

(E)

Three YearPersistence

Rate

(F)

Sig.Level

All 16.7 .01 15.6 .01 16.3 .01

Men 18.6 .01 16.3 .05 11.1 .10

Women 14.8 .01 15.9 .01 18.8 .01

Black 21.9 .01 18.9 .01 22.0 .05

Hispanic 17.0 .01 13.7 .05 15.4 .05

10

Work with what

you have.

Since 2011, 700+

people have been

trained in RBA in

Vermont.

What is Results-Based Accountability?

A framework for:

• evaluation,

• planning, and

• collaboration

...that is simple, common sense and useful.

RBA uses data to:

• inform decisions,

• challenge assumptions, and

• explain actions to stakeholders.

From Ends Means

From Talk Action

Result or OutcomeWhat do we want?

Indicator or BenchmarkHow we will recognize it?

What will it take to get there?

Performance MeasureHow will we know we made an impact?

Pop

ula

tio

nPe

rfo

rman

ce /

P

rogr

am

ENDS

MEANS

RBA’s 2 Levels of Accountability

Population (All of Vermont):• Vermont has a prosperous economy.

• Vermonters are healthy.

• Vermont’s children and young people achieve their

potential.

(E) Youth successfully transition to adulthood.

Performance (What we can be held to):• % of youth with positive skills acquired and changed

behavior

• % of students know about and can access community

resources related to careers and college

• % of students who are prepared for employment or post-

secondary education after completing a Navicate program

7 Qs for Population Accountability

1. What are the quality of life conditions we want for the

children, adults, and families who live in our

community?

2. What would these conditions look like if we could see

them?

3. How can we measure these conditions?

4. How are we doing on the most important of these

measures?

5. Who are the partners that have a role to play in doing

better?

6. What works to do better, including no-cost and low- cost

ideas?

7. What do we propose to do?

7 Qs for Performance Accountability

1. Who are our customers?

2. How can we measure if our customers are better off?

3. How can we measure if we are delivering services well?

4. How are we doing on the most important of these

measures?

5. Who are the partners that have a role to play in doing

better?

6. What works to do better, including no-cost and low- cost

ideas?

7. What do we propose to do?

Navicate Performance Examples

Result or Outcome• All Vermont youth are engaged, inspired, and equipped to achieve

their career and educational aspirations.

• All members of the Vermont community value and nurture youth in the

process, thus assuring the educational, economic, and social vitality of

the whole community.

Indicator or BenchmarkFor our students: Rate of high school graduation, percent pursuing post-

secondary education, teen unemployment rate, college persistence rate,

percent ready for careers

By Program: Performance Measures:Three Questions:

1. How much did we do?

2. How well did we do?

3. Is anyone better off?

Program Performance Measures

% of youth with positive skills acquired and

changed behavior

% of students know about and can access

community resources related to careers and

college

% of students who are prepared for employment or

post-secondary education after completing a

Navicate program

Is anyone better off? = Why we do our work

BEFORECollegeConnections

Disagree

AFTERMany more agree!

Contributionrelationship

Appropriateresponsibility

THE LINKAGE Between POPULATION and PERFORMANCE

POPULATION ACCOUNTABILITY

Youth are aware of and can find jobs.Rate of teen employment, rate of internships

Businesses are engaged in work-based learning. Gain of # of employers offering WBL opportunities to

youth

Youth are Pursuing Post-Secondary EducationPercent students pursuing post-secondary education,

rate of second year success

# ofstudents engaged

% of students successfully completing internships

# / % of students that know which school & community resources can help them

PERFORMANCE ACCOUNTABILITY

High School Internship Program

POPULATIONRESULTS

CUSTOMERRESULTS

Alignmentof measures

Comparing your performance

1. To ourselves -- can we do better than

our own history?

2. To others -- when it is a fair apples to

apples comparison

3. To standards -- when we know what

good performance is

Overview: Population and Performance

Accountability

All Youth in Vermont

WHOLE POPULATION

OUTCOME: Youth successfully transition

to adulthood

INDICATORSStudents have exposure and

access to real life and life-long learning

People involved in Navicate PERFORMANCE MEASURE

How much did we do?How well did we do it?Is anyone better off?

(i.e. Dual Enrollment,Internships,

Employers involved,Changed student skills/behaviors,VT youth engaged in school and

learning about themselves through careers and college

Navicate’s

Contribution

Means

End

RBA in a nutshell

2 levels of accountability

3 performance measures

7 accountability questions

from ends to means

Work with the data you

ALREADY have. Start

NOW!

Navicate’s main lesson:

How much did we do?

# of customers served

(by customer characteristic)

# activities

(by type of activity)

How well did we do it?

(% common measures)

workload ratio, staff turnover rate, staff morale,

percent of staff fully trained, worker safety, unit cost, customer satisfaction)

% activity-specific measures:

percent of actions timely and correct

percent of customers completing the activity

percent of actions meeting standards

IS ANYONE BETTER OFF? (# and %)

Skills and Knowledge

Attitude/Opinion

Behavior

Circumstance

EFFO

RT

EFFE

CT

QUANTITY QUALITY

Quantity (How Much We Do)

How much service did we

deliver?

Quality (How Well We Do It)

How well did we do it?

EFFORT

• 183 high school students served

• 204 employers engaged in

internships, tours, job-shadows,

career presentations etc.

• 4 high schools

• 1 alternative program providers

• 143 internships completed

• 87% successfully complete internships

(71 of 82 in 2013)

• 5% increase out of school youth served

• 8% increase internships completed (132

in 2012, 143 in 2013)

• 19 % increase of employers

(100 in 2012, 119 in 2013)

• 6% did not complete an internship

(9 of 143 in 2013)

RBA Grid

TIPS Internship Program – 1 yr. of data

Activity Time!

In your team, 1. Decide on one program to focus on2. Answer the three RBA questions:

- How much do we do?- How well do we do it?- Is anyone better off?

3. Fill in 3 parts of the blank RBA grid for that program.

Don’t get hung up over which category things go in – just get it on the paper and then discuss how it should be reported or which of the questions it answers.

Database

Grant language

Marketing

Performance Review of Organizational Health

Strategic Planning

Board Development

Staff Professional Development

A common language on how

to talk about our impact!

We even

made an

infographic.