7/27/2019 2013 Swc Beyond Toc 2

1/41

1

Beyond Theory of

Change

Tips and Strategies forCollecting, Aggregating, and

Reporting PM Outcomes

SW National Service Training ConferenceJuly 24, 2013

Susan Hyatt, Core Thought

www.core-thought.com

7/27/2019 2013 Swc Beyond Toc 2

2/41

2Session Description Knowing WHAT data to collect to demonstrate the

change your program creates is important.

However, data collection, aggregation and reporting toolsand processes are critical to insure you have valid and

reliable data on which to base your findings either makes

or breaks your efforts.

This session will provide you the step-wise information tohelp you insure your programs current performance

measurement approach is in alignment with your theory

of change and can yield the information you need to

clearly and confidently show your outcomes.

7/27/2019 2013 Swc Beyond Toc 2

3/41

3Learning Objectives

By the end of the session, you will:

Know the four types of outcomes and identify your optionsfor collecting data based on the outcome to be measured;Be able to identify the most appropriate data source and

cost effective data collection methodology to use to

measure your intended outcome; and

Know strategies for aggregating and summarizingperformance measurement data to make powerful

statements that make your program more attractive for

grants and other resources.

7/27/2019 2013 Swc Beyond Toc 2

4/41

4

Making the Case for

SupportCNCS funds program interventions

with a high likelihood of creating an

intended change in a communityneed

o Evidence-based interventions: Will

the proposed intervention (design,

dosage) likely lead to the desiredoutcome?

o Whats the cause and effect?

o What to measure to demonstrate

change?

7/27/2019 2013 Swc Beyond Toc 2

5/41

5

easur ng rogramPerformance

(Outcomes)

What is

Measure

-ment

Question?

Who

Would

Know?

TOC to

ClarifyDesiredOutcome How to

Collect

Info?

Then DoIt!

What

DoesInfo Tell

You? Tweak

Program

& Tell

YourStory

7/27/2019 2013 Swc Beyond Toc 2

6/41

6

Measure theAMOUNT of service

provided (Did you do what you said?)Number of people served (usually best)

Number of products/events/etc. created

What are Outputs?

Bean Counting

7/27/2019 2013 Swc Beyond Toc 2

7/41

7

Measure the CHANGES orBENEFITS

that occur because of service provided(individuals, organizations, communities, or theenvironment)

Increase - DecreaseMore - Less

Something is different

What are Outcomes?

The So What?

7/27/2019 2013 Swc Beyond Toc 2

8/41

8

Keep It Simple....You Cant

Measure EVERYTHING

Can read at grade

level

3rd Grade Reading Tutoring:

Whats the Best Outcome

to Measure?

Graduates from HS

7/27/2019 2013 Swc Beyond Toc 2

9/41

9

How Clear Are You

About What toMeasure?

Problem/Need

DesiredOutcome

ProgramActivities/Inte

rvention

Theory of Change:

Tool to Identify

Cause and Effect

Relationship

7/27/2019 2013 Swc Beyond Toc 2

10/41

10

What Are You Trying to

Change? (K-ABC)

KnowledgeSkill

Learning

new

information

or how to do

something

7/27/2019 2013 Swc Beyond Toc 2

11/41

11

What Are You Trying to

Change? (K-ABC)

AttitudeBelief

KnowledgeSkill

Learning

new

information

or how to do

something

Thoughts or

feelingsabout

something

7/27/2019 2013 Swc Beyond Toc 2

12/41

12

What Are You Trying to

Change? (K-ABC)

AttitudeBelief

KnowledgeSkill

Behavior

Learning

new

information

or how to do

something

Thoughts or

feelingsabout

something

Actions,

conduct,

or habits

7/27/2019 2013 Swc Beyond Toc 2

13/41

13

What Are You Trying to

Change? (K-ABC)

AttitudeBelief

KnowledgeSkill

Behavior Condition

Learning

new

information

or how to do

something

Thoughts or

feelingsabout

something

Actions,

conduct,

or habits

Circumstance

s or status

7/27/2019 2013 Swc Beyond Toc 2

14/41

14

What to Measure?

Teen Health Program

AttitudeBelief

KnowledgeSkill

Behavior Condition

Increased

knowledge of

healthy food

More willing

to eatvegetables

More frequent

exercise

Improved

physical

condition

7/27/2019 2013 Swc Beyond Toc 2

15/41

15

What Do TheseQuestions

Measure?IdentifyingOutputs and Outcome TypesACTIVITY

7/27/2019 2013 Swc Beyond Toc 2

16/41

16Now What?

You identified the desired change

(outcome) through your theory ofchange.

Are you clear about WHAT to measure?

AND

Are you measuring it the BEST way?

7/27/2019 2013 Swc Beyond Toc 2

17/41

17

Asking the Right

Question is CriticalIntervention: ABS driving school gives a 10 week

course that meets twice a week for 60 minutes that

includes classroom-based and on road lessons on

driving skills

Desired Outcome: Students have basic driving

proficiency.

7/27/2019 2013 Swc Beyond Toc 2

18/41

18

Asking the Right

Question

Question: Do you like driving?

Answer: I LOVE driving!!

Do you like driving? gets information about an

attitude not skill level.

7/27/2019 2013 Swc Beyond Toc 2

19/41

19

Asking the Right

Question

Question: Are you a skilled driver?

Answer: I think I am a GREAT driver!

Asking, Are you a skilled driver gets information

about self perception, a thought not actual skilllevel.

Self-ratings are subjective NOT objective.

7/27/2019 2013 Swc Beyond Toc 2

20/41

20

Asking the Right

Question

Question: Do you know the Colorado

driving laws?Answer: I got 100% correct on my

written drivers test!

Knowing state driving laws reflects knowledge

not actual skill level even though it is objective.

7/27/2019 2013 Swc Beyond Toc 2

21/41

21

Asking the Right

Question

Question: Have you demonstrated

you are a skilled driver?Answer: I passed my road test!

An on-road driving test DOES measure skill level or

proficiency and is objective.

7/27/2019 2013 Swc Beyond Toc 2

22/41

22Qualitative Data

Describes or characterizes through words

Focuses on meaning, experience or attitudesCollected through focus groups, interviews,

opened ended questionnaire items, and other

less structured situations.

Not the same as anecdotal Information

Qualitative Quality

7/27/2019 2013 Swc Beyond Toc 2

23/41

23Quantitative Data

Focus on numbers and frequencies.

Data which can be measuredLength, height, area, volume, weight, speed,

time, temperature, humidity, sound levels, cost,

ages, etc.

Quantitative Quantity

7/27/2019 2013 Swc Beyond Toc 2

24/41

24

Whats the Best Way to

Ask the Question?

Qualitative Quantitative

Please describe how you

have been feeling lately.

Thinking about the past week, how

depressed would you say you have

been on a scale from 0 to 5, where 0

means "not at all" and 5 means "the

most possible?

I feel great! I love my new job.

Ive lost 20 pounds and feel

healthier than I have in years.

2

7/27/2019 2013 Swc Beyond Toc 2

25/41

25

Whats Your Outcomeand Measurement

Question?ACTIVITY

7/27/2019 2013 Swc Beyond Toc 2

26/41

26

Whos Your Best

Source of Information? Data Source = The person, group or organization

that has information on whether the intended

outcome occurred.

Can you access this data source? How?

What are the pros and cons of relying on this data

source? What alternatives are available? What tradeoffs come

with relying on an alternative data source?

Once youve chosen an appropriate data source, make

sure youre asking them the right questions!

7/27/2019 2013 Swc Beyond Toc 2

27/41

27

School Attachment

Who Would Know?Students

(preferred data

source)

Teachers

Tutors

Parents

7/27/2019 2013 Swc Beyond Toc 2

28/41

28

Who is Your DataSource?

ACTIVITY

7/27/2019 2013 Swc Beyond Toc 2

29/41

29

What Method Will You

Use?

Data Collection Method: the process used to

systematically collect data

Match method with:

Measurement question for desired outcome

Data source

Resources (time, dollars, etc.)

7/27/2019 2013 Swc Beyond Toc 2

30/41

30Surveys

Types of Questions:

Open ended

Closed ended

Scales

RankingMultiple Choice

7/27/2019 2013 Swc Beyond Toc 2

31/41

31

Interviews & Focus

Groups

7/27/2019 2013 Swc Beyond Toc 2

32/41

32Observations

7/27/2019 2013 Swc Beyond Toc 2

33/41

33

Other Commonly Used

MethodsLogs and Tally Sheets

Standardized Tests

Program Developed Pre/Post Tests

Rubrics

Secondary Data

Other?

7/27/2019 2013 Swc Beyond Toc 2

34/41

34

Choosing or Designing

an Instrument

Which ever method you select, what

instrument will you use? Borrow vs. Develop

Does it ask the right questions to get at yourdesired outcome?

Does it have all the necessary components?

What information will each question yield?

How will you use information, if not related tooutcome?

7/27/2019 2013 Swc Beyond Toc 2

35/41

35Instrument MappingLook at each question on your data collection tool and

ask:

Does this help us measure the desired outcome? Is there one question? More than one?

What kind of data will we get? Subjective? Objective?

Quantitative? Qualitative? If it doesnt measure the outcome, do we really need to ask

it? How will we use the answer? Nice to know or internal

use?

How will we analyze this?

7/27/2019 2013 Swc Beyond Toc 2

36/41

36

Instrument Mapping

ACTIVITY

7/27/2019 2013 Swc Beyond Toc 2

37/41

37Data Collection StepsAfter identifying a data source, method and

instrument(s):

1.Identify who you need to work with to collect data

2.Set a schedule for collecting data

3.Train your data collectors

4.Implement data collection

Good practice is to pilot test your tool.

FOR BEST RESULTS make key decis ion s about how to

imp lement data col lect ion BEFORE your p rogram b egins!

7/27/2019 2013 Swc Beyond Toc 2

38/41

38Program PM TargetsTarget describes amount of change you anticipate

achieving. Based on indicator specific, measurable

amount of change that will signify success

Apply target to your specific instrument(s)

If instruments vary by site - which questions get at

the desired outcome?

Need to apply program target to data collection toolsused by each site

Need to report aggregate outcome for program - not

each site separately

7/27/2019 2013 Swc Beyond Toc 2

39/41

39

Data Aggregation andReporting

ACTIVITY

7/27/2019 2013 Swc Beyond Toc 2

40/41

40Summary of Key Points Use your theory of change as a guide to outcome

measurement.

The type of outcome you want to measure (attitude,knowledge, behavior, condition) influences your

choice of data source, method, and instrument.

Be very clear about the measurement question to

ask Choose methods based on ability to measure your

desired outcome and ease of use.

Map your instrument

Remember: Continuous Improvement NOT

7/27/2019 2013 Swc Beyond Toc 2

41/41

41Ways to Connect

Susan Hyattshyatt@core-thought.com

303-512-0808

@Susan_Hyatt

http://www.facebook.com/CoreThoughtinc

www.core-thought.com

mailto:shyatt@core-thought.comhttp://www.core-thought.com/http://www.core-thought.com/http://www.core-thought.com/http://www.core-thought.com/mailto:shyatt@core-thought.commailto:shyatt@core-thought.commailto:shyatt@core-thought.com